Recently in geomechanics, much attention has been paid not only to the quality of geomechanical laboratory tests but also to maintaining the quality of the core from which the samples are drilled. During core drilling, two types of impact on the core can be defined: Mechanical impact of the drill head to break the rock and form the core; change in the load at the boundaries of the core and wellbore, which is determined by the pressure of the drilling fluid. In this work, a numerical algorithm has been developed and implemented in 3D code, which simulates the creation of new surfaces in the process of core recovery and setting on them boundary conditions corresponding to the pressure of the drilling fluid, as well as the effect of the drilling tool. On its basis, a numerical study of deformation and fracture processes during core recovery for a real layered medium of an unconventional reservoir was performed.