The concept of physics-informed neural networks has become a useful tool for solving differential equations due to its flexibility. A few approaches use this concept to solve the eikonal equation that describes the first-arrival traveltimes of waves propagating in smooth heterogeneous velocity models. However, the challenge of the eikonal is exacerbated by the velocity models producing caustics, resulting in instabilities and deterioration of accuracy due to the non-smooth solution behavior. In this paper, we revisit the problem of solving the eikonal equation using neural networks to tackle caustic pathologies. We introduce the novel Neural Eikonal Solver (NES) for solving the isotropic eikonal equation in two formulations: the one-point problem is for a fixed source location; the two-point problem is for an arbitrary source-receiver pair. We present several techniques which provide stability in the case of caustics: improved factorization; non-symmetric loss function based on Hamiltonian; gaussian activation; symmetrization. In our tests, NES showed the relative mean-absolute error of 0.2-0.4% from the second-order factored Fast Marching Method with a similar inference time, and outperformed existing neural-network solvers giving 10-60 times lower errors and 2-30 times faster training. The one-point NES provides the most accurate solution, while the two-point NES gives slightly lower accuracy but an extremely compact representation with all spatial derivatives. It can be useful in many seismic problems: massive computations of traveltimes for millions of source-receiver pairs in Kirchhoff migration; modeling of ray amplitudes using spatial derivatives; traveltime tomography; earthquake localization; ray multipathing analysis.