Help with GE ultrasound implementation

[nicxo99]

Hello!

I am trying to connect a GE ultrasound machine to the Plus framework for free-hand calibration (fCal). I am using 3D Slicer to check that the data is valid. I wrote a small program in C++ that sends image data using OpenIGTLink. The image data is originally captured using GE AppAPI interface. Using this program, I am successfully able to stream directly to Slicer using the OpenIGTLinkIF module:

bool StreamLink::sendImage(const cv::Mat& frame, uint64_t timestamp_ms)
{
    if (socket_.IsNull())
        return false;

    cv::Mat img = frame.clone();

    // Convert BGR -> RGB
    int channels = img.channels();
    if (channels == 3)
    {
        cv::cvtColor(img, img, cv::COLOR_BGR2RGB);
    }

    //----------------------------------
    // IMAGE SETUP
    //----------------------------------

    int size[3] = { img.cols, img.rows, 1 };

    auto imageMsg = igtl::ImageMessage::New();

    imageMsg->SetHeaderVersion(IGTL_HEADER_VERSION_1);

    imageMsg->SetDimensions(size);

    int svsize[3] = { img.cols, img.rows, 1 };
    int svoffset[3] = { 0, 0, 0 };

    imageMsg->SetSubVolume(svsize, svoffset);

    float spacing[3] = { 1.0f, 1.0f, 1.0f };
    imageMsg->SetSpacing(spacing);

    imageMsg->SetNumComponents(channels);
    imageMsg->SetScalarType(igtl::ImageMessage::TYPE_UINT8);

    igtl::Matrix4x4 matrix;
    igtl::IdentityMatrix(matrix);

    matrix[1][1] = -1;
    matrix[1][3] = img.rows - 1;

    imageMsg->SetMatrix(matrix);

    //----------------------------------
    // Coordinate system
    //----------------------------------

    imageMsg->SetCoordinateSystem(igtl::ImageMessage::COORDINATE_LPS);

    //----------------------------------
    // Device name
    //----------------------------------

    imageMsg->SetDeviceName("Video");

    //----------------------------------
    // Allocate image buffer
    //----------------------------------

    imageMsg->AllocateScalars();

    memcpy(
        imageMsg->GetScalarPointer(),
        img.data,
        img.total() * img.elemSize()
    );

    //----------------------------------
    // Timestamp
    //----------------------------------

    igtl::TimeStamp::Pointer ts = igtl::TimeStamp::New();

    uint64_t sec = timestamp_ms / 1000;
    uint64_t nsec = (timestamp_ms % 1000) * 1000000;

    ts->SetTime(sec, nsec);
    imageMsg->SetTimeStamp(ts);

    static uint32_t frameNumber = 0;
    frameNumber++;
    imageMsg->SetMetaDataElement("FrameNumber", frameNumber);

    //----------------------------------
    // Pack + send
    //----------------------------------

    imageMsg->Pack();

    int sent = socket_->Send(
        imageMsg->GetPackPointer(),
        imageMsg->GetPackSize()
    );

    //std::cout << "Sent frame "
    //    << img.cols << "x" << img.rows
    //    << " channels=" << channels
    //    << " bytes=" << imageMsg->GetPackSize()
    //    << std::endl;

    return sent > 0;
}

However, when integrating it with the Plus framework I can connect to my OpenIGTLink server, but I am not able to receive the data in Slicer anymore when going through the Plus Server interface. I am using this config file in the PlusToolkit:

<PlusConfiguration version="2.9">

  <DataCollection StartupDelaySec="1.0">
    <DeviceSet
      Name="fCal: GE ultrasound + Atracsys"
      Description="Broadcasting ultrasound data from GE HealthCare + tracking data from Atracsys spryTrack180, fusionTrack250 or fusionTrack500 through OpenIGTLink."
    />
    <Device
      Id="TrackerDevice"
      Type="AtracsysTracker"
      MaxMissingFiducials="1"
      MaxMeanRegistrationErrorMm="1.0"
      ToolReferenceFrame="Tracker" >
      <DataSources>
        <DataSource Type="Tool" Id="Stylus" TrackingType="PASSIVE" GeometryFile="AtracsysTools/geometry420.ini" GeometryId="420" />
        <DataSource Type="Tool" Id="Probe" TrackingType="PASSIVE" GeometryFile="AtracsysTools/geometry505.ini" GeometryId="505" />
      </DataSources>
      <OutputChannels>
        <OutputChannel Id="TrackerStream">
        <DataSource Type="Tool" Id="Stylus" />
        <DataSource Type="Tool" Id="Probe" />
        </OutputChannel>
      </OutputChannels>
    </Device>
	
	<!-- ========================= -->
    <!-- ULTRASOUND (FROM C++ IGTL SERVER) -->
    <!-- ========================= -->
    <Device
      Id="VideoDevice"
      Type="OpenIGTLinkVideo"
	  MessageType="IMAGE"
      ServerAddress="127.0.0.1"
      ServerPort="18945"
      Active="true"
	  IgtlMessageCrcCheckEnabled="false"
      LocalTimeOffsetSec="0"
	  ReceiveTimeoutSec="5"
	  ReconnectOnReceiveTimeout="true">

      <DataSources>
        <DataSource Id="Video" Type="Video" PortUsImageOrientation="MF" />
      </DataSources>

      <OutputChannels>
        <OutputChannel Id="VideoStream"
                       VideoDataSourceId="Video"/>
      </OutputChannels>

    </Device>

    <!-- ========================= -->
    <!-- FUSION (TRACKED VIDEO) -->
    <!-- ========================= -->
    <Device
      Id="TrackedVideoDevice"
      Type="VirtualMixer"
	  ToolReferenceFrame="Tracker" >

      <InputChannels>
        <InputChannel Id="TrackerStream"/>
        <InputChannel Id="VideoStream"/>
      </InputChannels>

      <OutputChannels>
        <OutputChannel Id="TrackedVideoStream"/>
      </OutputChannels>

    </Device>

  </DataCollection>

  <!-- ========================= -->
  <!-- OPENIGTLINK SERVER (TO SLICER) -->
  <!-- ========================= -->
  <PlusOpenIGTLinkServer
    ListeningPort="18944"
    OutputChannelId="TrackedVideoStream"
    SendValidTransformsOnly="TRUE">

    <DefaultClientInfo>
      <MessageTypes>
        <Message Type="IMAGE"/>
        <Message Type="TRANSFORM"/>
      </MessageTypes>
    </DefaultClientInfo>

    <TransformNames>
      <Transform Name="StylusToTracker"/>
      <Transform Name="ProbeToTracker"/>
    </TransformNames>
	
	<ImageNames>
        <Image Name="Video" EmbeddedTransformToFrame="Probe" />
      </ImageNames>

  </PlusOpenIGTLinkServer>
  
</PlusConfiguration>

And I receive these errors in the Plus Server log, which indicate that maybe something is wrong with my timestamps, I just don’t know what. Maybe you can figure out why I get these errors, or what I need to change in order to make it work:

052226_123756.913|INFO|011.913000| Received new client connection (client 1 at 127.0.0.1:18944). Number of connected clients: 1| in D:\a\PlusLib\PlusLib\build\PlusLib\src\PlusServer\vtkPlusOpenIGTLinkServer.cxx(298)
052226_123756.927|TRACE|011.927000| vtkPlusAtracsysTracker::InternalUpdate| in D:\a\PlusLib\PlusLib\build\PlusLib\src\PlusDataCollection\Atracsys\vtkPlusAtracsysTracker.cxx(716)
052226_123756.927|INFO|011.927000| OpenIGTLink broadcasting started. No data was available between 0-5.514sec, therefore no data were broadcasted during this time period.| in D:\a\PlusLib\PlusLib\build\PlusLib\src\PlusServer\vtkPlusOpenIGTLinkServer.cxx(449)
052226_123756.927|TRACE|011.927000| vtkPlusDevice::GetTrackedFrameList(5.614, 50)| in D:\a\PlusLib\PlusLib\build\PlusLib\src\PlusDataCollection\vtkPlusChannel.cxx(719)
052226_123756.927|TRACE|011.927000| Unable to get latest timestamp from video buffer!| in D:\a\PlusLib\PlusLib\build\PlusLib\src\PlusDataCollection\vtkPlusChannel.cxx(1275)
052226_123756.927|TRACE|011.927000| Failed to get video buffer item UID from time: 11.911000| in D:\a\PlusLib\PlusLib\build\PlusLib\src\PlusDataCollection\vtkPlusChannel.cxx(1427)
052226_123756.927|TRACE|011.927000| Failed to get video buffer timestamp from UID: 268| in D:\a\PlusLib\PlusLib\build\PlusLib\src\PlusDataCollection\vtkPlusChannel.cxx(1429)
052226_123756.927|TRACE|011.927000| Unable to get most recent timestamp!| in D:\a\PlusLib\PlusLib\build\PlusLib\src\PlusDataCollection\vtkPlusChannel.cxx(756)
052226_123756.927|TRACE|011.927000| Number of added frames: 50 out of 50| in D:\a\PlusLib\PlusLib\build\PlusLib\src\PlusDataCollection\vtkPlusChannel.cxx(937)
052226_123756.934|TRACE|011.935000| Failed to get tracked frame list from data collector (last recorded timestamp: 11.863000| in D:\a\PlusLib\PlusLib\build\PlusLib\src\PlusServer\vtkPlusOpenIGTLinkServer.cxx(454)