Oil fired boiler fault. Running of it's own accord!?

I've got oil fired central heating and the boiler decides to fire up of it's own accord when the room thermostat is turned down or even off. Heard it fire up this morning, went downstairs & thermostat was at 5 degrees (room was about 15) so no need for it to kick in. Checked all the radiators and only 2 of them were hot, and others luke warm. Have turned it off at the mains electric and lit the fire instead - I wondered why the last lot of kerosene dwindled so quickly! Any ideas? Checked manual and can only find info on "boiler not firing" as opposed to "boiler firing constantly!" Thanks