Learning shared template representation with augmented feature for multi-object pose estimation.

Template matching pose estimation methods based on deep learning have made significant advancements via metric learning or reconstruction learning. Existing approaches primarily build distinct template representation libraries (codebooks) from rendered images for each object, which complicate the training process and increase memory cost for multi-object tasks. Additionally, they struggle to effectively handle discrepancies between the distributions of training and test sets, particularly for occluded objects, resulting in suboptimal matching accuracy. In this study, we propose a shared template representation learning method with augmented semantic features to address these issues. Our method learns representations concurrently using metric and reconstruction learning as similarity constraints, and augments response of network to objects through semantic feature constraints for better generalization performance. Furthermore, rotation matrices serve as templates for codebook construction, leading to excellent matching accuracy compared to rendered images. Notably, it contributes to the effective decoupling of object categories and templates, necessitating the maintenance of only a shared codebook in multi-object pose estimation tasks. Extensive experiments on Linemod, Linemod-Occluded and TLESS datasets demonstrate that the proposed method employing shared templates achieves superior matching accuracy. Moreover, proposed method exhibits robustness on a collected aircraft dataset, further validating its efficacy.

Calibration method for large-field-of-view stereo vision system based on distance-related distortion model.

Large-field-of-view stereo vision system lacks flexible and high-precision calibration methods. To this end, we proposed a new distance-related distortion model based calibration method combining 3D points and checkerboards. The experiment indicates that the proposed method has a root mean square of fewer than 0.08 pixels for the reprojection error on the calibration dataset, and the mean relative error of length measurement in a volume of 5.0 m × 2.0 m × 16.0 m is 3.6‰. Compared with other distance-related models, the proposed model has the lowest reprojection error on the test dataset. Besides, in contrast to other calibration methods, our method offers enhanced accuracy and greater flexibility.

Calibration method of spatial transformations between the non-orthogonal two-axis turntable and its mounted camera.

In order to expand the field of view and measuremenst range, the camera is often mounted on a two-axis turntable to perform various visual tasks. And the calibration of the position and attitude relationship between the mounted camera and the two-axis turntable is a prerequisite for visual measurement. The turntable is considered an ideal orthogonal two-axis turntable in conventional methods. However, the rotation axes of the actual two-axis turntable may be neither vertical nor intersecting, and the optical center of the mounted camera is not always located in the rotation center of the turntable even for orthogonal two-axis turntables. The quite difference between the actual physical model of the two-axis turntable and the ideal model can cause large errors. Therefore, what we believe to be a novel position and attitude calibration method between a non-orthogonal two-axis turntable and the mounted camera is proposed. This method describes the spatial hetero-planar lines relationship between the azimuth axis and pitch axis of the turntable accurately. By the geometric invariant characteristics of the mounted camera in motion, the axes of turntable are recovered and the base coordinate system is established, and the position and attitude of the camera are calibrated. Simulation and experiments verify the correctness and effectiveness of our proposed method.

Stratified camera calibration algorithm based on the calibrating conic.

In computer vision, camera calibration is essential for photogrammetric measurement. We propose a new stratified camera calibration method based on geometric constraints. This paper proposes several new theorems in 2D projective transformation: (1) There exists a family of lines whose parallelity remains invariable in a 2D projective transformation. These lines are parallel with the image of the infinity line. (2) There is only one line whose verticality is invariable with the family of parallel lines in a 2D projective transformation, and the principal point lies on this line. With the image of the infinite line and the dual conic of the circular points, the closed-form solution of the line passing through principal point is deduced. The angle among the target board and image plane, which influences camera calibration, is computed. We propose a new geometric interpretation of the target board's pose and solution method. To obtain appropriate poses of the target board for camera calibration, we propose a visual pose guide (VPG) of the target board system that can guide a user to move the target board to obtain appropriate images for calibration. The expected homography is defined, and its solution method is deduced. Experimental results with synthetic and real data verify correctness and validity of the proposed method.

Target-aware transformer tracking with hard occlusion instance generation.

Visual tracking is a crucial task in computer vision that has been applied in diverse fields. Recently, transformer architecture has been widely applied in visual tracking and has become a mainstream framework instead of the Siamese structure. Although transformer-based trackers have demonstrated remarkable accuracy in general circumstances, their performance in occluded scenes remains unsatisfactory. This is primarily due to their inability to recognize incomplete target appearance information when the target is occluded. To address this issue, we propose a novel transformer tracking approach referred to as TATT, which integrates a target-aware transformer network and a hard occlusion instance generation module. The target-aware transformer network utilizes an encoder-decoder structure to facilitate interaction between template and search features, extracting target information in the template feature to enhance the unoccluded parts of the target in the search features. It can directly predict the boundary between the target region and the background to generate tracking results. The hard occlusion instance generation module employs multiple image similarity calculation methods to select an image pitch in video sequences that is most similar to the target and generate an occlusion instance mimicking real scenes without adding an extra network. Experiments on five benchmarks, including LaSOT, TrackingNet, Got10k, OTB100, and UAV123, demonstrate that our tracker achieves promising performance while running at approximately 41 fps on GPU. Specifically, our tracker achieves the highest AUC scores of 65.5 and 61.2% in partial and full occlusion evaluations on LaSOT, respectively.

On-Orbit Calibration of Installation Matrix between Remote Sensing Camera and Star Camera Based on Vector Angle Invariance.

To achieve photogrammetry without ground control points (GCPs), the precise measurement of the exterior orientation elements for the remote sensing camera is particularly important. Currently, the satellites are equipped with a GPS receiver, so that the accuracy of the line elements of the exterior orientation elements could reach centimeter-level. Furthermore, the high-precision angle elements of the exterior orientation elements could be obtained through a star camera which provides the direction reference in the inertial coordinate system and star images. Due to the stress release during the launch and the changes of the thermal environment, the installation matrix is variable and needs to be recalibrated. Hence, we estimate the cosine angle vector invariance of a remote sensing camera and star camera which are independent of attitude, and then we deal with long-term on-orbit data by using batch processing to realize the accurate calibration of the installation matrix. This method not only removes the coupling of attitude and installation matrix, but also reduces the conversion error of multiple coordinate systems. Finally, the geo-positioning accuracy in planimetry is remarkably higher than the conventional method in the simulation results.

A Line Matching Method Based on Multiple Intensity Ordering with Uniformly Spaced Sampling.

This paper presents a line matching method based on multiple intensity ordering with uniformly spaced sampling. Line segments are extracted from the image pyramid, with the aim of adapting scale changes and addressing fragmentation problem. The neighborhood of line segments was divided into sub-regions adaptively according to intensity order to overcome the difficulty brought by various line lengths. An intensity-based local feature descriptor was introduced by constructing multiple concentric ring-shaped structures. The dimension of the descriptor was reduced significantly by uniformly spaced sampling and dividing sample points into several point sets while improving the discriminability. The performance of the proposed method was tested on public datasets which cover various scenarios and compared with another two well-known line matching algorithms. The experimental results show that our method achieves superior performance dealing with various image deformations, especially scale changes and large illumination changes, and provides much more reliable correspondences.

High-accuracy calibration of line-structured light vision sensors using a plane mirror.

A high-accuracy calibration technique using a white paper and a front coating plane mirror is proposed in this paper for line-structured light vision sensors. This method shows advantages in two aspects. First, a white paper can gain a very high-quality light stripe due to its approximate ideal diffuse Lambertian sheet, which overcomes the problems associated with the strong reflecting light and serious burrs of the light stripe on conventional rigid targets. Second, based on a front coating plane mirror with lithographic feature points, we can obtain a bilateral symmetric structure similar to a virtual binocular stereo vision to recover the 3D coordinates of the light stripe centers on white paper with high accuracy. Front coating guarantees the coplanarity with the lithographic feature points and avoids imaging distortion caused by refraction during back coating. Therefore, front coating can be used to obtain high accuracy structural parameters of the virtual binocular stereo vision sensors. Meanwhile, for the light stripe and its image in the plane mirror are auto-epipolar with all the epipolar lines arranged in parallel. These lines intersect at a vanishing point in the camera image, and this epipolar constraint is used to complete the matching of the light stripe centers without the need for the camera parameters. Experiments are conducted to demonstrate the performance of the proposed method.

Interactive Manipulation of Nonconductive Microparticles in Scanning Electron Microscope by a Virtual Nano-hand Strategy.

Micro/nano-manipulation is the fabrication of particular constructs on devices at the micro/nano-scale. Precise manipulation of microparticles is one of the key technological difficulties in manufacturing micro/nano-scale components. Based on scanning electron microscopy and nanomanipulator, this paper adopts a direct push method to operate randomly distributed microparticles into ordered structures. A two-probe interaction strategy is proposed to enable microparticle movements in all directions efficiently and avoid scratching the substrate surface. To overcome the uncertainties in micromanipulation, a virtual nano-hand strategy was also implemented: long-range advance of each microparticle is realized by multiple single-step pushes, whose trajectory is theoretically analyzed. The pushes are well programmed to imitate effects of a more powerful and determined hand. Experimental results show that the theoretical single-step motion trajectory is in line with actual operation, and the proposed strategy can ensure precise operation of the microparticles in all directions and improve reliability and effectiveness of operation.

Extrinsic parameters calibration of multi-camera with non-overlapping fields of view using laser scanning.

An extrinsic parameters calibration method of multi-cameras with non-overlapping fields of view (FOV) using laser scanning is presented. Firstly, two lasers are mounted on a multi-degree-of-freedom manipulator and can scan objects freely by the projected line-structured light. Then, controlling the movement of the manipulator, the line-structured light is projected into the field of view of one of the multi-cameras, and the light plane equation in the camera coordinate frame is calibrated by the target. The manipulator is moved several times in small amplitude to change the position of structured light in the field of vision of the camera and to continue to calibrate the light plane. The light plane equation of line-structured light in the manipulator coordinate frame are solved by the hand-eye calibration method. Secondly, with the help of the light planes, projected into the field of vision of other cameras to be calibrated, the light plane equation in the camera coordinate frame is calibrated, and the external parameters between the camera coordinate frame and the manipulator coordinate frame are calculated, so that the calibration of the external parameters of multiple cameras can be realized. The proposed method connects the non-overlapping multi-cameras by the laser scanning. It can effectively solve the problem of multi-camera extrinsic parameter calibration under the conditions of long working distance and complex environment light.

Angular distance constraints calibration for outdoor zoom camera.

Based on 2-D protractor property of camera, we proposed a flexible calibration method for zoom camera that used outdoors. It only requires the camera to observe control points once for given zooming settings, when there are several control points at infinity and known the angular distances. Under constraints of image points, the angular distance between their re-projecting vectors and the image of absolute conic (IAC), nonlinear optimization is used to solve parameters of IAC. Then IAC can be uniquely decomposed by the Cholesky factorization, and consequently the intrinsic parameters can be obtained. Towards the factors that affect the accuracy of the calibration, theoretical analysis and computer simulation are carried out respectively consequence in qualitative analysis and quantitative result. On the issues of inaccuracy of principal point, the zooming center is selected to improve the accuracy of calibration. Real data demonstrated the effectiveness of the techniques.

Global calibration of unleveled theodolite using angular distance constraints.

The theodolite is an important optical measurement instrument in application. Its global calibration, including position and orientation, is a prerequisite for measurement. Most global calibration methods require the theodolite to be leveled precisely, which is time-consuming and susceptible to error. We propose a global calibration method without leveling: it solves position results using the angular distance of control points by nonlinear optimization and then computes orientation parameters (rotation matrix) linearly based on position results. Furthermore, global calibration of multiple theodolites is also introduced. In addition, we introduced a method that can compute the dip direction and tilt angle by decomposing the rotation matrix. We evaluate the calibration algorithms on both computer simulation and real data experiments, demonstrating the effectiveness of the techniques.

Structural Parameters Calibration for Binocular Stereo Vision Sensors Using a Double-Sphere Target.

Structural parameter calibration for the binocular stereo vision sensor (BSVS) is an important guarantee for high-precision measurements. We propose a method to calibrate the structural parameters of BSVS based on a double-sphere target. The target, consisting of two identical spheres with a known fixed distance, is freely placed in different positions and orientations. Any three non-collinear sphere centres determine a spatial plane whose normal vector under the two camera-coordinate-frames is obtained by means of an intermediate parallel plane calculated by the image points of sphere centres and the depth-scale factors. Hence, the rotation matrix R is solved. The translation vector T is determined using a linear method derived from the epipolar geometry. Furthermore, R and T are refined by nonlinear optimization. We also provide theoretical analysis on the error propagation related to the positional deviation of the sphere image and an approach to mitigate its effect. Computer simulations are conducted to test the performance of the proposed method with respect to the image noise level, target placement times and the depth-scale factor. Experimental results on real data show that the accuracy of measurement is higher than 0.9‰, with a distance of 800 mm and a view field of 250 × 200 mm².

Real-Time Visual Tracking through Fusion Features.

Due to their high-speed, correlation filters for object tracking have begun to receive increasing attention. Traditional object trackers based on correlation filters typically use a single type of feature. In this paper, we attempt to integrate multiple feature types to improve the performance, and we propose a new DD-HOG fusion feature that consists of discriminative descriptors (DDs) and histograms of oriented gradients (HOG). However, fusion features as multi-vector descriptors cannot be directly used in prior correlation filters. To overcome this difficulty, we propose a multi-vector correlation filter (MVCF) that can directly convolve with a multi-vector descriptor to obtain a single-channel response that indicates the location of an object. Experiments on the CVPR2013 tracking benchmark with the evaluation of state-of-the-art trackers show the effectiveness and speed of the proposed method. Moreover, we show that our MVCF tracker, which uses the DD-HOG descriptor, outperforms the structure-preserving object tracker (SPOT) in multi-object tracking because of its high-speed and ability to address heavy occlusion.

Vanishing feature constraints calibration method for binocular vision sensor.

Based on analyzing the measurement model of binocular vision sensor, we proposed a new flexible calibration method for binocular vision sensor using a planar target with several parallel lines. It only requires the sensor to observe the planar target at a few (at least two) different orientations. Relying on vanishing feature constraints and spacing constraints of parallel lines, linear method and nonlinear optimization are combined to estimate the structure parameters of binocular vision sensor. Linear method achieves the separation of the rotation matrix and translation vector which reduces the complexity of computation; Nonlinear algorithm ensures the calibration results for the global optimization. Towards the factors that affect the accuracy of the calibration, theoretical analysis and computer simulation are carried out respectively consequence in qualitative analysis and quantitative result. Real data shows that the accuracy of the proposed calibration method is about 0.040mm with the working distance of 800mm and the view field of 300 × 300mm. The comparison with Bougust toolbox and the method based on known length indicates that the proposed calibration method is precise and is efficient and convenient as its simple calculation and easy operation, especially for onsite calibration and self-calibration.

Correction method for line extraction in vision measurement.

Over-exposure and perspective distortion are two of the main factors underlying inaccurate feature extraction. First, based on Steger's method, we propose a method for correcting curvilinear structures (lines) extracted from over-exposed images. A new line model based on the Gaussian line profile is developed, and its description in the scale space is provided. The line position is analytically determined by the zero crossing of its first-order derivative, and the bias due to convolution with the normal Gaussian kernel function is eliminated on the basis of the related description. The model considers over-exposure features and is capable of detecting the line position in an over-exposed image. Simulations and experiments show that the proposed method is not significantly affected by the exposure level and is suitable for correcting lines extracted from an over-exposed image. In our experiments, the corrected result is found to be more precise than the uncorrected result by around 45.5%. Second, we analyze perspective distortion, which is inevitable during line extraction owing to the projective camera model. The perspective distortion can be rectified on the basis of the bias introduced as a function of related parameters. The properties of the proposed model and its application to vision measurement are discussed. In practice, the proposed model can be adopted to correct line extraction according to specific requirements by employing suitable parameters.

Microscopic vision based on the adaptive positioning of the camera coordinate frame.

Microscopic vision measurement precision has been largely limited by inaccurately calibrated model parameters, because image plane is near parallel to reference plane in the narrow depth of field. This article proposes a method of precise microscopic vision measurement based on the adaptive positioning of the camera coordinate frame. The microscopic vision measurement movably attaches the origin of the camera coordinate frame along the optical axis. By finding the optimal position, the nonlinearity of the objective function in calibration optimization is decreased and the optimization sensitivity to initial values is reduced. Therefore, we obtain a high calibration precision and eventually ensure a high measurement precision. Mathematical simulations illustrate that the calibration precision of the proposed microscopic vision measurement model is higher than that of the conventional vision measurement model. The experiment shows that with magnification of 3.024×, the presented system achieves a precision of 0.12% based on the proposed microscopic vision measurement model, which is two times higher than the one based on the conventional vision measurement model.