Before you take that one off you need to figure out what you will replace it with. An original Krag sight will a real project. Butlers Rangers has posted a photo of an original barrel with the sight base unsoldered revealing the dove tail.
The dovetail cutter that was used is long lost as far as dimensions and such. A cutter would have to be custom made. Then the real work would begin.
A front sight adapted from the 1903 Springfield is seen often so it would not look too out of place.
If you have access to a milling machine the existing sight base could be milled to resemble an original Krag if care was taken to not knock it off during machining.
There were other after market sights made back in the Krag's heyday that you might be able find today.
Others with advice will want to know what rear sight is on the rifle.