The rapid development of mixed reality (MR) technology has opened up new possibilities for remote collaborative tasks in assembly scenarios. As the preparatory work of the MR collaboration system, physical virtualization is the main task of providing reliable model support and the relationship between the virtual and the real for user browsing and interaction. The existing methods cannot take into account the goals of high detail reduction and low virtualization cost of assembly parts at the same time. A precision recovery method of part model based on the principle of template matching and point cloud alignment is proposed to realize the replacement of the reconstructed point cloud model in the original scene using a high precision part CAD model. For the task related parts in the process of MR remote collaboration, firstly, the prior information related to the part point cloud is obtained through the segmentation of background point cloud based on background difference and octree spatial retrieval and the segmentation of foreground adhesive point cloud based on supervoxel to segment the ROI of the region of interest of template matching, which solves the problem of weak occlusion resistance and low template comparison efficiency of the original LineMod algorithm, and the identification matching and rough estimation of the part point cloud poses are completed. The estimated poses were then further optimized using the ICP algorithm to achieve positional recovery of the CAD model in the scene. Through the physical virtualization experiments of various parts in complex assembly scene, it is proved that this method can accurately identify the point cloud of parts and realize accurate physical virtualization, which has important practical value in MR remote collaborative task.