There are only 2 real options for mobile phone companies in Libya: Al Madar and Libyana. Many Libyans actually have 2 SIM cards, one for each company.
Currently you can purchase a new SIM card for 5 LYD from each of these companies around the cities. You will be required to give them a copy of your passport and sign a contract in your name.
At some points in time, SIM card sales have been stopped. If you are not able to buy a SIM card in from the companies, there are shops who sell used SIM cards. If you buy a used SIM card, make sure you get the contract signed over to you so that it will be officially yours.
If you are looking to use 3G on your phone, make sure to have them activate it when you sign up.