Normally you compile the entire kernel, not just one module.
Edit: If you know what you are doing, you can build the module “out of tree”. See Barry’s reply below.
There is some documentation for doing that on Fedora Linux, but I have not tried to use it in ages, so I cannot speak to how up-to-date the documentation is.