There are things I'd like to blog about nearly every week. This week I thought I'd actually get around to doing it, about a simple thing that really should have clearer instructions elsewhere online. We're going to use the Log4Net package to log information in an output file, in an ASP.NET MVC controller. For this example, I used log4net version 2.
First, using the NuGet package manager, install the package for log4net into your project.
Put this line in the file AssemblyInfo.cs:
[assembly: log4net.Config.XmlConfigurator(Watch = true)]
Put this line in the file global.asax:
log4net.Config.XmlConfigurator.Configure();
Put this in the web.config file:
<configSections>
...
<section name="log4net" type="log4net.Config.Log4NetConfigurationSectionHandler, log4net" />
</configSections>
<log4net debug="true">
<appender name="RollingLogFileAppender" type="log4net.Appender.RollingFileAppender">
<file value="Logs/log.log" />
<appendToFile value="true" />
<rollingStyle value="Size" />
<maxSizeRollBackups value="10" />
<maximumFileSize value="100KB" />
<staticLogFileName value="true" />
<layout type="log4net.Layout.PatternLayout">
<conversionPattern value="%-5p %d %5rms %-22.22c{1} %-18.18M - %m%n" />
</layout>
</appender>
<root>
<level value="DEBUG" />
<appender-ref ref="RollingLogFileAppender" />
</root>
</log4net>
Rebuild the solution/project.
Put the logging commands in a class method, e.g. a controller:
using log4net;
public class MyController : Controller {
ILog log = LogManager.GetLogger(typeof(MyController));
log.Debug("* Debug message");
log.Warn("* Warning message");
log.Error("* Error message");
log.Info("* Information message");
}
Run the application (push F5).
Check the project root for a new folder and a log file, named "log.log". Done!
Monday, 18 September 2017
Saturday, 24 June 2017
TimeSpans, Ticks and BigInts for C#, SQL Server & NHibernate
I'm writing this post about the data type for C# called TimeSpan. The gist of this article is that it's easy to create a TimeSpan in C# and get information about it, but it's tougher to do the same thing for storing its "ticks" units using SQL in a SQL Server database field.
A TimeSpan is not a complicated concept, you may think. It stores the number of units of time up to a point in a certain time range, usually within one day. The units are called "ticks". Here's my example, creating a TimeSpan for the time on a day of 2:30pm.
TimeSpan tSpan = new TimeSpan(14, 30, 00);
Console.WriteLine("Ticks in the TimeSpan: " + tSpan.Ticks);
The output is: 522000000000, i.e. that many ticks have occurred since the start of the time range (midnight, or 00:00:00).
Now, some of you may have the "privilege" of using the SQL ORM for C#, NHibernate. You'll notice that what NHibernate does for storing TimeSpans in a SQL Server database is convert them into BIGINT numbers. The reason for this, I'm guessing, is that the number of ticks can end up very large for representing a TimeSpan.
If you do a SELECT query on a database table which you've mapped through NHibernate, you would see that number of ticks stored in there when you save your save your entity and execute your session transaction (i.e. ticks is 522,000,000,000).
Now, the key to converting a C# TimeSpan to a BIGINT with SQL is this next number. Once I knew it, it was plain sailing:
The number of ticks in one day = 864,000,000,000
To help you deal with that total-ticks-in-one-day figure, you also need to calculate the number of seconds in one day:
The number of seconds in one day = 86,400
(ie. 60 seconds x 60 minutes x 24 hours)
Now, when you calculate the total ticks in one day divided by the total seconds in one day, you get this number:
The number of ticks per second = 10,000,000
(i.e. 864000000000 ticks / 86400 seconds)
So, from there, stop and think. If you can get your TimeSpan's hours, minutes and seconds, all you will need to do to turn this into a total number of ticks is:
e.g. for my 2:30pm TimeSpan:
((14 * 60 * 60) + (30 * 60) + 0 * 10000000
= 522000000000
That figure is that same as the one NHibernate stored. We're on the right track! So, let's turn this into a Transact-SQL statement, for inserting a record with those calculations in it. I'm going to declare my hour, minutes and seconds values as variables beforehand.
GO
declare @myHours int = 14;
declare @myMinutes int = 30;
declare @mySeconds int = 0;
declare @ticksPerSecond bigint = 10000000;
INSERT INTO [dbo].[MyTable]
(
[TimeOfDay]
)
VALUES
(
((@myHours * 60 * 60) + (@myMinutes * 60) + (@mySeconds)) * @ticksPerSecond
)
GO
My desired result (of 522,000,000,000) for the TimeOfDay field appears in the database correctly. T-SQL stores the value as a BIGINT in the same way that NHibernate stores a C# TimeSpan.
A TimeSpan is not a complicated concept, you may think. It stores the number of units of time up to a point in a certain time range, usually within one day. The units are called "ticks". Here's my example, creating a TimeSpan for the time on a day of 2:30pm.
TimeSpan tSpan = new TimeSpan(14, 30, 00);
Console.WriteLine("Ticks in the TimeSpan: " + tSpan.Ticks);
The output is: 522000000000, i.e. that many ticks have occurred since the start of the time range (midnight, or 00:00:00).
Now, some of you may have the "privilege" of using the SQL ORM for C#, NHibernate. You'll notice that what NHibernate does for storing TimeSpans in a SQL Server database is convert them into BIGINT numbers. The reason for this, I'm guessing, is that the number of ticks can end up very large for representing a TimeSpan.
If you do a SELECT query on a database table which you've mapped through NHibernate, you would see that number of ticks stored in there when you save your save your entity and execute your session transaction (i.e. ticks is 522,000,000,000).
Now, the key to converting a C# TimeSpan to a BIGINT with SQL is this next number. Once I knew it, it was plain sailing:
The number of ticks in one day = 864,000,000,000
To help you deal with that total-ticks-in-one-day figure, you also need to calculate the number of seconds in one day:
The number of seconds in one day = 86,400
(ie. 60 seconds x 60 minutes x 24 hours)
Now, when you calculate the total ticks in one day divided by the total seconds in one day, you get this number:
The number of ticks per second = 10,000,000
(i.e. 864000000000 ticks / 86400 seconds)
So, from there, stop and think. If you can get your TimeSpan's hours, minutes and seconds, all you will need to do to turn this into a total number of ticks is:
- convert each figure's value into a total number of seconds
- multiple the sum of all these seconds values by 10,000,000
e.g. for my 2:30pm TimeSpan:
((14 * 60 * 60) + (30 * 60) + 0 * 10000000
= 522000000000
That figure is that same as the one NHibernate stored. We're on the right track! So, let's turn this into a Transact-SQL statement, for inserting a record with those calculations in it. I'm going to declare my hour, minutes and seconds values as variables beforehand.
GO
declare @myHours int = 14;
declare @myMinutes int = 30;
declare @mySeconds int = 0;
declare @ticksPerSecond bigint = 10000000;
INSERT INTO [dbo].[MyTable]
(
[TimeOfDay]
)
VALUES
(
((@myHours * 60 * 60) + (@myMinutes * 60) + (@mySeconds)) * @ticksPerSecond
)
GO
My desired result (of 522,000,000,000) for the TimeOfDay field appears in the database correctly. T-SQL stores the value as a BIGINT in the same way that NHibernate stores a C# TimeSpan.
Friday, 9 October 2015
Crawling with OpenWebSpider
Spiderman, Where are You Coming From Spiderman?
Sometimes you just need the right tools to help you do what you want. This is especially true when the tool you are limited to has a bunch of restrictions. I ran into this particular problem the other day when I wanted to "spider" a website to find out all its possible URLs being served, including the URLs that are ignored by the usual "robots.txt" file sitting on the web server.I actually had to find a way around the "robots.txt" file for a web site that was blocking web crawlers from finding most of the URLs that were available. Imagine a spider that is crawing through a tunnel, looking for all the tasty bugs - and there is a giant boulder in the way providing the biggest obstacle possible to getting those bugs it wants to feed on.
My requirements for a spider alternative (if possible) were:
- free
- no lengthy configuration
- a GUI
- a way of ignoring the "robots.txt" file's instructions
- the search results in a spreadsheet file
- able to run on Windows (and Linux too, if possible)
OpenWebSpider is a neat application written in node.js, which crawls a website and saves its results to a database that you nominate. The project seems to be reasonably active and at the time of writing is hosted on SourceForge. To get the application going in Windows, first you'll need to do a few things.
Install WAMP Server
OK, so the database I wanted to store my results in was MySQL. One of the best ways to get this (and other nice bits) in Windows is to install the handy WAMP server package. (However, the providers do warn you that you might first need a Visual Studio C++ 2012 redistributable package installed.)One you do get WAMP Server installed, you'll have a nice new "W" icon in your system tray. Click on it, then click Start All Services.
You can view the MySQL database in the phpMyAdmin interface, just to be nice and friendly. In your browser then, go to:
http://localhost/phpmyadmin
Note: if you have any troubles with phpMyAdmin, you might have to edit its config file, usually in this location:
C:\wamp\apps\phpmyadmin4.x\config.inc.php
In phpMyAdmin, create a database to store your spidering results (I've named my database "ows"). Also create a user for the database.
Install node.js
Node.js is one of the most wonderful Javascript frameworks out there, and it's what OpenWebSpider is written with. So go to the node.js project site, download the installer and run it. Check that node.js is working OK by opening a command shell window and typing the word "node".
Get OpenWebSpider and Create the Database's Schema
Download the OWS zip file from Sourceforge. Unzip the file. Inside the project folder, you'll see a number of files. Double-click the "openwebspider.bat" file to launch the application in a little shell window.
Then, as the readme.txt instruction file tells you:
- Open a web-browser at http://127.0.0.1:9999/
- Go in the third tab (Database) and configure your settings
- Verify that openwebspider correctly connects to your server by clicking the "Verify" button
- "Save" your configuration
- Click "Create DB"; this will create all tables needed by OpenWebSpider
Now, check your database's new tables in phpMyAdmin:
Start Spidering!
Go to the OWS browser view, ie. at http://127.0.0.1:9999/. Click on the Worker tab and alter any settings you might thing useful. Enter the URL of the site you want to crawl in the URL box (and make sure "http://www" is in front if needed). Then hit the Go! button.
View Your Results
In phpMyAdmin, click on the pages table and you should automatically see a view of the crawling results. You might not need all the columns you'll see, in fact my usual SQL query to run is just something like:select hostname, page, title from pages where hostname = 'www.website.com';
And of course, anyone with half a brain knows you can export data from phpMyAdmin as a CSV file. Then you can view your data by importing the CSV file into a spreadsheet application.
Ignore Those Pesky Robots
Please note, the next instruction is for the version of OpenWebSpider from October 2015. It probably is outdated for the latest version now.One of the requirements was that we could bypass the "robots.txt" file, which good web crawlers by default must follow. What you'll need to do is close OpenWebSpider, and just edit one source code file. Find this file and open it up in a text editor: openwebspider\src\_worker\_indexerMixin.js.
It's a Javascript file, so all you need to do is comment out these lines (with // symbols):
// if (!canFetchPage || that.stopSignal === true)
// {
// if (!canFetchPage)
// {
// msg += "\n\t\t blocked by robots.txt!";
// }
// else
// {
// msg += "\n\t\t stop signal!";
// }
// logger.log("index::url", msg);
// callback();
// return;
// }
Re-save the file. Then launch OWS again, and try another search. You'll usually find the number of found URL results has gone up, since the crawler is now ignoring the instructions in the "robots.txt" file.
Everyday Use
It's a good idea to make a desktop shortcut for the "openwebspider.bat" file. So, if you want to use OpenWebSpider regularly, what you'll need to remember to do each time you need it is:- Start up the WAMP server in the system tray
- On the desktop, double-click the "openwebspider.bat" shortcut icon
- In a browser, go to localhost:9999 (Openwebspider) to run the worker
- In a browser, go to localhost/phpmyadmin to see your results
Happy spidering!
Saturday, 28 February 2015
Python & Google Search API Requests
It's been a while since I've been able to blog something. Anyway, out of all the geeky things I could write about which I've been tinkering with (reponsive HTML email layouts, Symfony2 secure users set ups, etc)., I just thought I'd share something for a Google API that should have easy instructions for accessing it through Python somewhere but doesn't. You'll see lots of information out there about putting a CustomSearch box and results in your browser with Javascript block, but we want to do it with Python, right?
Google provide a generic "how to access Google APIs through Python" guide here:
https://developers.google.com/api-client-library/python/
However, it's not too helpful in showing you specifically how to just get Google search results out, which is what we want! It also doesn't tell you that at the time of writing Google haven't yet made a version of their module for Python 3.
Anyway, the background seems to be that years ago Google had a rather clunky XML/SOAP interface for making requests using an old Search API. Thankfully they got rid of it and replaced with a much nicer JSON API. There is a free version of this new one called the CustomSearch API, which limits you to 100 requests per day. (If you want more there is a paid version.) The catch for this is that you are also limited to only ten search results per request. I guess Google unhelpfully assume you will want to paginate search results, with ten results per page? So for example, if you want to get 100 results searching for something, you'll have to call the CustomSearch API ten times (i.e. 10 requests = 10 x 10 search results).
OK, so anyway it's Google you're dealing with. That means, if you want to use their CustomSearch API, you need a Google account. So if you haven't created a Google account yet, go ahead and create one. Then what you'll need to do is:
(a) create a Project and assign a Google API to it
(b) create a Search Engine with an ID
Go to the Google Developers Console and login:
https://console.developers.google.com
Click the "Create Project" button
-Enter a Project Name, e.g. "testSearch1"
-Enter a Project ID (Google will suggest one for you)
Click the "Enable an API" button
-click on the Custom Search API
Select "Credentials" in the left navigation bar under "APIs & auth"
Under Public API access, click on Create new Key
Then click >Server Key >Create (don't bother entering anything in optional IP addresses).
Copy the key code it gives you and save it somewhere safe.
Next, go to the CustomSearch Search Engine management console page. You need to go here, because you need a search engine and its ID to be able to do Google search requests.
https://www.google.com/cse/manage/all
Click on >New search engine
-Enter a Search Engine name, e.g. Google
-Enter a site to search, e.g. www.google.com
-Choose Search the entire web but emphasize included sites
-After this is finished, in the Edit screen for the search engine you've created, click on the Search Engine ID button. Copy this ID code, you'll need it for later.
Check the available Google APIs pages and find the details for the CustomSearch API. You'll notice it has one entity: cse. This is what we're going to focus on.
https://developers.google.com/apis-explorer/#p/
https://developers.google.com/resources/api-libraries/documentation/customsearch/v1/python/latest/customsearch_v1.cse.html
Have a look in there and you will see a large number of parameters we can pass in to the entity when doing a request via the API. The only parameters we really need though are:
q - the search query string
cx - the search engine ID we created
start - the offset from the beginning of the search results.
OK, our goal is write a quick script in Python to run on the command line. It will make a number of requests using the Google CustomSearch API and send the results to JSON output file. (I'm assuming you're using Python 2.x and have pip installed to download your Python packages.)
Install the Google API client module:
pip install -U google-api-python-client
Edit this program below, inserting your own Project ID and Search Engine ID. I've saved it as "search_google.py", and made a new directory in my current directory named "output" first. This is where the results file will be outputted.
#!/usr/bin/env python
import datetime as dt
import json, sys
from apiclient.discovery import build
if __name__ == '__main__':
# Create an output file name in the format "srch_res_yyyyMMdd_hhmmss.json"
now_sfx = dt.datetime.now().strftime('%Y%m%d_%H%M%S')
output_dir = './output/'
output_fname = output_dir + 'srch_res_' + now_sfx + '.json'
search_term = sys.argv[1]
num_requests = int(sys.argv[2])
# Key codes we created earlier for the Google CustomSearch API
search_engine_id = '[My Search Engine ID]'
api_key = '[My Project API Key]'
# The build function creates a service object. It takes an API name and API
# version as arguments.
service = build('customsearch', 'v1', developerKey=api_key)
# A collection is a set of resources. We know this one is called "cse"
# because the CustomSearch API page tells us cse "Returns the cse Resource".
collection = service.cse()
output_f = open(output_fname, 'ab')
for i in range(0, num_requests):
# This is the offset from the beginning to start getting the results from
start_val = 1 + (i * 10)
# Make an HTTP request object
request = collection.list(q=search_term,
num=10, #this is the maximum & default anyway
start=start_val,
cx=search_engine_id
)
response = request.execute()
output = json.dumps(response, sort_keys=True, indent=2)
output_f.write(output)
print('Wrote 10 search results...')
output_f.close()
print('Output file "{}" written.'.format(output_fname))
Run the program on the command line with the format:
python search_google.py "[my search term]" [number of requests x 10]
e.g.
python search_google.py "vintage tiddlywinks" 3
Your JSON file should be outputted in the output directory you created, containing your search results.
Google provide a generic "how to access Google APIs through Python" guide here:
https://developers.google.com/api-client-library/python/
However, it's not too helpful in showing you specifically how to just get Google search results out, which is what we want! It also doesn't tell you that at the time of writing Google haven't yet made a version of their module for Python 3.
Anyway, the background seems to be that years ago Google had a rather clunky XML/SOAP interface for making requests using an old Search API. Thankfully they got rid of it and replaced with a much nicer JSON API. There is a free version of this new one called the CustomSearch API, which limits you to 100 requests per day. (If you want more there is a paid version.) The catch for this is that you are also limited to only ten search results per request. I guess Google unhelpfully assume you will want to paginate search results, with ten results per page? So for example, if you want to get 100 results searching for something, you'll have to call the CustomSearch API ten times (i.e. 10 requests = 10 x 10 search results).
Shut Up & Show Me the Instructions
OK, so anyway it's Google you're dealing with. That means, if you want to use their CustomSearch API, you need a Google account. So if you haven't created a Google account yet, go ahead and create one. Then what you'll need to do is:
(a) create a Project and assign a Google API to it
(b) create a Search Engine with an ID
(a) Create a Project and assign a Google API to it
Go to the Google Developers Console and login:
https://console.developers.google.com
Click the "Create Project" button
-Enter a Project Name, e.g. "testSearch1"
-Enter a Project ID (Google will suggest one for you)
Click the "Enable an API" button
-click on the Custom Search API
Select "Credentials" in the left navigation bar under "APIs & auth"
Under Public API access, click on Create new Key
Then click >Server Key >Create (don't bother entering anything in optional IP addresses).
Copy the key code it gives you and save it somewhere safe.
(b) create a Search Engine with an ID
Next, go to the CustomSearch Search Engine management console page. You need to go here, because you need a search engine and its ID to be able to do Google search requests.
https://www.google.com/cse/manage/all
Click on >New search engine
-Enter a Search Engine name, e.g. Google
-Enter a site to search, e.g. www.google.com
-Choose Search the entire web but emphasize included sites
-After this is finished, in the Edit screen for the search engine you've created, click on the Search Engine ID button. Copy this ID code, you'll need it for later.
Check the available Google APIs pages and find the details for the CustomSearch API. You'll notice it has one entity: cse. This is what we're going to focus on.
https://developers.google.com/apis-explorer/#p/
https://developers.google.com/resources/api-libraries/documentation/customsearch/v1/python/latest/customsearch_v1.cse.html
Have a look in there and you will see a large number of parameters we can pass in to the entity when doing a request via the API. The only parameters we really need though are:
q - the search query string
cx - the search engine ID we created
start - the offset from the beginning of the search results.
Let's get it going with Python
OK, our goal is write a quick script in Python to run on the command line. It will make a number of requests using the Google CustomSearch API and send the results to JSON output file. (I'm assuming you're using Python 2.x and have pip installed to download your Python packages.)
Install the Google API client module:
pip install -U google-api-python-client
Edit this program below, inserting your own Project ID and Search Engine ID. I've saved it as "search_google.py", and made a new directory in my current directory named "output" first. This is where the results file will be outputted.
#!/usr/bin/env python
import datetime as dt
import json, sys
from apiclient.discovery import build
if __name__ == '__main__':
# Create an output file name in the format "srch_res_yyyyMMdd_hhmmss.json"
now_sfx = dt.datetime.now().strftime('%Y%m%d_%H%M%S')
output_dir = './output/'
output_fname = output_dir + 'srch_res_' + now_sfx + '.json'
search_term = sys.argv[1]
num_requests = int(sys.argv[2])
# Key codes we created earlier for the Google CustomSearch API
search_engine_id = '[My Search Engine ID]'
api_key = '[My Project API Key]'
# The build function creates a service object. It takes an API name and API
# version as arguments.
service = build('customsearch', 'v1', developerKey=api_key)
# A collection is a set of resources. We know this one is called "cse"
# because the CustomSearch API page tells us cse "Returns the cse Resource".
collection = service.cse()
output_f = open(output_fname, 'ab')
for i in range(0, num_requests):
# This is the offset from the beginning to start getting the results from
start_val = 1 + (i * 10)
# Make an HTTP request object
request = collection.list(q=search_term,
num=10, #this is the maximum & default anyway
start=start_val,
cx=search_engine_id
)
response = request.execute()
output = json.dumps(response, sort_keys=True, indent=2)
output_f.write(output)
print('Wrote 10 search results...')
output_f.close()
print('Output file "{}" written.'.format(output_fname))
Run the program on the command line with the format:
python search_google.py "[my search term]" [number of requests x 10]
e.g.
python search_google.py "vintage tiddlywinks" 3
Your JSON file should be outputted in the output directory you created, containing your search results.
Saturday, 3 May 2014
Eclipse Kepler and PDT
The other day I found myself reinstalling stuff on my laptop and one of the things I wanted to put on there was my favourite programming IDE, Eclipse. It had a been a year since I'd used it frequently, and I had to follow a number of steps to get it going properly with PDT (PHP Development Tools), the PHP development plugin. I was surprised how little information I could find quickly online to help with doing this; so I thought I'd share the steps I'd followed.
I carried this out in Ubuntu Linux 14.04, but I think some of the things should work similarly in another OS like Windows if you tried. I'm assuming you've installed things for a LAMP environment already, i.e. Apache web server and the PHP language. I am also going to stick with the default XDebug debugger for PHP as I find it fine, even though the Zend Debugger is trendy at the moment.
In your browser, go to the Oracle site:
http://www.oracle.com/technetwork/java/javase/downloads
Choose Java Platform, Standard Edition > Download >JDK
Choose Accept >Linux x64 >jdk-8u5-linux-x64.tar.gz
(the JDK you choose may vary based on version number and your processor. It's about 150 mb in size).
In a shell, make yourself a directory for the Java Virtual Machine. Copy your downloaded JDK there and unzip it. (The version placeholder below will hold your version number, e.g. "1.8.0_05"):
cd [my download directory];
mkdir /usr/lib/jvm;
cp jdk[version].tar.gz /usr/lib/jvm;
cd /usr/lib/jvm;
tar xvf jdk[version].tar.gz;
rm jdk[version].tar.gz;
Find your Java compiler and update your system. Set the permissions how you like.
ls /usr/lib/jvm/jdk[version]/bin/;
-You will see the Java compiler there, i.e. see "javac". Note this down.
chmod -R 770 /usr/lib/jvm/jdk[version];
update-alternatives --install /usr/bin/javac javac /usr/lib/jvm/jdk[version]/bin/javac 1;
update-alternatives --install /usr/bin/java java /usr/lib/jvm/jdk[version]/bin/java 1;
update-alternatives --config javac;
update-alternatives --config java;
Check all is OK:
java -version
The output should be something like:
java version "1.8.0_05"
Java(TM) SE Runtime Environment (build 1.8.0_05-b13)
Java HotSpot(TM) 64-Bit Server VM (build 25.5-b02, mixed mode)
http://www.eclipse.org/downloads/
Copy your download over to handy directory, like /opt. Then unzip it and set permissions:
cd [my download directory];
cp eclipse-standard-kepler-[version].tar.gz /opt
cd /opt;
tar xvf eclipse-standard-kepler-[version].tar.gz
rm eclipse-standard-kepler-[version].tar.gz
sudo chown -R [my user]:root eclipse;
sudo chmod -R +r eclipse;
Put Eclipse in your user PATH, e.g:
sudo ln -s /opt/eclipse/eclipse /usr/bin/eclipse
Start Eclipse on the command line with:
eclipse -clean &
Or make item in Ubuntu's launcher menu (with the Alacarte application):
sudo apt-get install alacarte
alacarte &
Choose >Programming >Add Item
name: eclipse4.3.2
path: /opt/eclipse/eclipse
Help >Install new software >Add
For name, type: Kepler updates
for Location, type: http://download.eclipse.org/releases/kepler
You will now see an install plugin wizard
-Choose >Programming Languages >PHP Development Tools (PDT) > Next
-You will see a Review >Next
-Choose "I Accept" >Finish
-Then you will see install progress bar
You can check PDT is installed by going to:
-Window >Open Perspective >Other >PHP
-This switches to the PHP Development perspective
Set up some editor preferences how you like:
-Go to Window >Preferences >PHP >Code Style >Formatter
-Set Profile to PHP Conventions, choose >Show
-Go to >Indentation tab
-Tab policy: spaces
-Indentation size: 2
-go to >Line wrapping: max line width = 80
Set your PHP executable. If you don't know this, in a shell type "which php".
-Go to Window >Preferences >PHP Executables >Add
-Name: php559
-Exec: /usr/bin/php
-php ini: /etc/php5/apache2/php.ini
-php debugger: XDebug
Set up console output with the debugger:
-Go to Window >Preferences >PHP >Debug - choose XDebug
-Choose >Enable CLI debugging
Note: PDT is not set up well for console output - it opens a separate debug tab, which can be annoying. So to turn this off:
Go to Window >Preferences >Run/Debug >Perspectives
For the 2 lines under "PHP CLI application":
- Choose Debug at right
- For "Open the associated perspective when launching", choose Never (turns off auto switch to Debug tab)
- For "Open the associated perspective when an application suspends", choose "Prompt"
To actually run a PHP script to get command line output, what I had to do was:
-select my opened PHP file in Eclipse
-right-click to see the options for "Run as PHP CLI Application" / "Debug as CLI Application"
(the menu at top doesn't seem to populate with these options)
or push Alt-Shift-X H.
Window >Preferences >JavaScript >Code Style >Formatter
-Choose >New profile, and name a new one, e.g: Eclipse_custom_by_me
-I changed the tab settings for this editor:
-Choose > Edit > Indentation
-Choose Spaces Only, 2, 2 >OK
>OK
-Go to: Help >Install new software >Add
-For Name, type: Eclipse Color Theme
-For Location, type: http://eclipse-color-theme.github.com/update
-Choose Select >Next, and you will see a progress bar
-Choose >Next >I Accept >Finish
Eclipse will then Restart. Choose your theme by going to:
Window >Preferences >General >Appearance >Color Theme
(My favourite theme is Oblivion. It's nice and dark.)
Now you're ready to write the next world-beating PHP project in Eclipse using the PDT plugin!
I carried this out in Ubuntu Linux 14.04, but I think some of the things should work similarly in another OS like Windows if you tried. I'm assuming you've installed things for a LAMP environment already, i.e. Apache web server and the PHP language. I am also going to stick with the default XDebug debugger for PHP as I find it fine, even though the Zend Debugger is trendy at the moment.
Step 1: Give Me Full-Strength Java
Eclipse is built in Java, so that's what you need first. I am not a fan of the open source Java versions (they're kind like a decaffeinated cup of coffee), so I downloaded and installed the official Oracle Java Development Kit.In your browser, go to the Oracle site:
http://www.oracle.com/technetwork/java/javase/downloads
Choose Java Platform, Standard Edition > Download >JDK
Choose Accept >Linux x64 >jdk-8u5-linux-x64.tar.gz
(the JDK you choose may vary based on version number and your processor. It's about 150 mb in size).
In a shell, make yourself a directory for the Java Virtual Machine. Copy your downloaded JDK there and unzip it. (The version placeholder below will hold your version number, e.g. "1.8.0_05"):
cd [my download directory];
mkdir /usr/lib/jvm;
cp jdk[version].tar.gz /usr/lib/jvm;
cd /usr/lib/jvm;
tar xvf jdk[version].tar.gz;
rm jdk[version].tar.gz;
Find your Java compiler and update your system. Set the permissions how you like.
ls /usr/lib/jvm/jdk[version]/bin/;
-You will see the Java compiler there, i.e. see "javac". Note this down.
chmod -R 770 /usr/lib/jvm/jdk[version];
update-alternatives --install /usr/bin/javac javac /usr/lib/jvm/jdk[version]/bin/javac 1;
update-alternatives --install /usr/bin/java java /usr/lib/jvm/jdk[version]/bin/java 1;
update-alternatives --config javac;
update-alternatives --config java;
Check all is OK:
java -version
The output should be something like:
java version "1.8.0_05"
Java(TM) SE Runtime Environment (build 1.8.0_05-b13)
Java HotSpot(TM) 64-Bit Server VM (build 25.5-b02, mixed mode)
Step 2: Install Eclipse
Go the Eclipse site and download Eclipse Standard 4.3.x:http://www.eclipse.org/downloads/
Copy your download over to handy directory, like /opt. Then unzip it and set permissions:
cd [my download directory];
cp eclipse-standard-kepler-[version].tar.gz /opt
cd /opt;
tar xvf eclipse-standard-kepler-[version].tar.gz
rm eclipse-standard-kepler-[version].tar.gz
sudo chown -R [my user]:root eclipse;
sudo chmod -R +r eclipse;
Put Eclipse in your user PATH, e.g:
sudo ln -s /opt/eclipse/eclipse /usr/bin/eclipse
Start Eclipse on the command line with:
eclipse -clean &
Or make item in Ubuntu's launcher menu (with the Alacarte application):
sudo apt-get install alacarte
alacarte &
Choose >Programming >Add Item
name: eclipse4.3.2
path: /opt/eclipse/eclipse
Step 3: Install PDT (the PHP development plugin)
In Eclipse, go to:Help >Install new software >Add
For name, type: Kepler updates
for Location, type: http://download.eclipse.org/releases/kepler
You will now see an install plugin wizard
-Choose >Programming Languages >PHP Development Tools (PDT) > Next
-You will see a Review >Next
-Choose "I Accept" >Finish
-Then you will see install progress bar
You can check PDT is installed by going to:
-Window >Open Perspective >Other >PHP
-This switches to the PHP Development perspective
Set up some editor preferences how you like:
-Go to Window >Preferences >PHP >Code Style >Formatter
-Set Profile to PHP Conventions, choose >Show
-Go to >Indentation tab
-Tab policy: spaces
-Indentation size: 2
-go to >Line wrapping: max line width = 80
Set your PHP executable. If you don't know this, in a shell type "which php".
-Go to Window >Preferences >PHP Executables >Add
-Name: php559
-Exec: /usr/bin/php
-php ini: /etc/php5/apache2/php.ini
-php debugger: XDebug
CLI My Output
Now you come to a point where you decide whether: (a) you want to run things from Eclipse and direct them to a web server/browser, or (b) you want to view command line output. I like option (b), especially for testing.Set up console output with the debugger:
-Go to Window >Preferences >PHP >Debug - choose XDebug
-Choose >Enable CLI debugging
Note: PDT is not set up well for console output - it opens a separate debug tab, which can be annoying. So to turn this off:
Go to Window >Preferences >Run/Debug >Perspectives
For the 2 lines under "PHP CLI application":
- Choose Debug at right
- For "Open the associated perspective when launching", choose Never (turns off auto switch to Debug tab)
- For "Open the associated perspective when an application suspends", choose "Prompt"
To actually run a PHP script to get command line output, what I had to do was:
-select my opened PHP file in Eclipse
-right-click to see the options for "Run as PHP CLI Application" / "Debug as CLI Application"
(the menu at top doesn't seem to populate with these options)
or push Alt-Shift-X H.
Step 4: Configure some extras in Eclipse
PDT'S Javascript Editor
The PDT seems to come with a helpful Javascript Editor. You can configure this by going to:Window >Preferences >JavaScript >Code Style >Formatter
-Choose >New profile, and name a new one, e.g: Eclipse_custom_by_me
-I changed the tab settings for this editor:
-Choose > Edit > Indentation
-Choose Spaces Only, 2, 2 >OK
>OK
The Eclipse Color Theme plugin
One thing I don't like is a glaring white screen, and this plugin lets you configure a number of different themes for Eclipse that are much friendlier to your eyes. To install it:-Go to: Help >Install new software >Add
-For Name, type: Eclipse Color Theme
-For Location, type: http://eclipse-color-theme.github.com/update
-Choose Select >Next, and you will see a progress bar
-Choose >Next >I Accept >Finish
Eclipse will then Restart. Choose your theme by going to:
Window >Preferences >General >Appearance >Color Theme
(My favourite theme is Oblivion. It's nice and dark.)
Sunday, 23 June 2013
Raspberry Pi fun
It's been way too long since I made a blog post. Last December I was given a Raspberry Pi model B as a present from my brother-in-law, and I've finally got together and done some things with it.
This is the list of things I already had (besides the Pi):
To make a long story short, as there are lots of blogs out there already on what people have done, I got the Raspbian OS installed on the SD card and everything running through an ethernet internet connection displayed on the old CRT television. Afterwards, I hooked the wifi adapter, HP printer and USB hard drive to the self-powered USB brick. Then:
It didn't really cost much to get a little low-powered device going with these useful services. I'm hoping that using a Pi instead of a laptop will save some money off my power bills :-D
This is the list of things I already had (besides the Pi):
- an old 14" CRT TV
- a yellow RCA video cable
- a 3.5mm stereo cable
- a USB keyboard
- a USB mouse
- a Seagate 2TB external self-powered HDD
- an HP PSC1310 wired USB printer
- a Linksys ADSL2+ router
- a 16GB SD card ($NZ20)
- a 5v power adapter ($NZ7)
- a self-powered 7 port USB brick ($NZ7.50)
- 2 USB2.0 female/male 2-metre extension cords ($NZ13.80)
- an ASUS WL-167G V3 USB wifi adapter ($NZ23)
To make a long story short, as there are lots of blogs out there already on what people have done, I got the Raspbian OS installed on the SD card and everything running through an ethernet internet connection displayed on the old CRT television. Afterwards, I hooked the wifi adapter, HP printer and USB hard drive to the self-powered USB brick. Then:
1) Network savvy
I edited the /etc/network/interfaces file to use my wifi adapter for a wireless connection, and assigned a static IP address to it. I also installed samba so that Windows computers in my home network could access the Pi.2) Connecting remotely
I installed the putty program on my Windows 7 laptop, so that I could log in by ssh console to the pi using the static IP address. Once I was in, I turned off the GUI mode by using raspi-config (why waste resources on a little Pi device?)3) File server / file shares
I set up a file share for others in my home network. I also mapped the networked USB drive connected to the Raspberry Pi to a Windows drive (Z:):4) Printer server
Then I got the wired printer working with the CUPS printer service on the Pi. I had to change the CUPS config file and check the settings in the CUPS web interface. Then on my Windows laptop, I had to add the networked printer and choose the right drivers for it:5) Torrent device
Lastly, I wanted to use the pi as a low-powered torrent downloading device, and store what it downloaded on the USB HDD. I was already using Deluge as a torrent client in Windows, which I liked. I found out there is a Windows thin client available to connect deluge to a network Linux machine running a deluged daemon. So I had to install deluged and its helpful deluge-console on the Pi. Then I could use the Windows desktop Deluge client to connect to the deluged daemon on the Pi.It didn't really cost much to get a little low-powered device going with these useful services. I'm hoping that using a Pi instead of a laptop will save some money off my power bills :-D
Saturday, 24 November 2012
Django, uwsgi, nginx and virtualenv
Bring on the Pain - just kidding!
There are lots of tutorials online for getting the Python Django web framework and the nginx web server working together. However, some of them skip over specific details of what to do and automatically assume you're a Linux God, able to configure anything hi-tech using only your little finger. Because of this, I've decided to create my own tutorial for doing this with Python 2.7x and a virtualenv.
One part of getting things to work together is the pain of dealing with uWSGI. uWSGI calls itself an "application container server coded in pure C", which deals with the protocols for communicating between the web server (nginx) and your Python framework (Django). What it really is, in fact, is a great WSGI interface written by talented hackers which is, however, rather user-unfriendly.
In addition, most documentation for Django for using uwsgi seems slightly out of date, and uwsgi's own documentation is skimpy and really only hints at what you can do, without practically helping much, especially for those of us whose skills don't include hardcore Linux administration. For example, it took me ages to realize that the options to put in a uwsgi ini file are the same as those displayed when you type "uwsgi --help". Also, I have only a basic knowledge of Unix sockets, and so I got quite stuck.
So I thought, I'll address this! Here is a quick, get-you-up-and-running guide for deploying with Django, uwsgi, nginx and a virtualenv. (This is not a guide to teach you the basics of any of those four things. Only to configure them so you get a web app running!) This guide was tested out in Mint 12 and Ubuntu Server 12.04. It is for a Django project "progress_recorder" that will be stored in my /var/www/py directory - you would replace the name and location with those of your own project.
Steps
1) Install the pip and virtualenv packages for Python
sudo apt-get install python-pip
pip install virtualenv
2) Make a virtualenv for a Python version (and activate)
cd /opt
virtualenv py273
source /opt/py273/bin/activate
3) Install Django into your virtualenv
pip install django
# Check version
python -c "import django; print(django.get_version())"
1.4.2
4) Install uwsgi into your virtualenv
pip install uwsgi
# Check version
/opt/py273/bin/uwsgi --version
1.4.1
5) Install nginx
sudo apt-get install nginx
# Check version
nginx -V
nginx: nginx version: nginx/1.1.19
# Check it's running
service nginx start
ps aux | grep nginx
- you should see "nginx: master process" in the output somewhere.
6) Make a Django project
-Get the page views/templates all displaying with the test server (python manage.py runserver).
Here is the structure of my project, named "progress_recorder", in a tree outline:
progress_recorder
├── manage.py
└── progress_recorder
├── django.ini
├── django_wsgi.py
├── __init__.py
├── settings.py
├── templates
│ ├── entry.html
│ ├── index.html
│ └── __init__.py
├── urls.py
└── views
├── entry.py
├── index.py
├── __init__.py
6.a) Set your templates in settings.py
TEMPLATE_DIRS = (
'/var/www/py/progress_recorder/progress_recorder/templates',)
6.b) Set your url patterns in urls.py
urlpatterns = patterns('',
url(r'^progress_recorder/entry$', 'views.entry.entry'),
url(r'^progress_recorder$', 'views.index.index'),
)
7) Get Django working with uwsgi
7.a) Make a django_wsgi.py file:
vim django_wsgi.py
#!/usr/bin/env python
import os
import django.core.handlers.wsgi
os.environ.setdefault("DJANGO_SETTINGS_MODULE", "progress_recorder.settings")
application = django.core.handlers.wsgi.WSGIHandler()
7.b) Make a uwsgi ini file, named django.ini:
vim django.ini
[uwsgi]
pythonpath = /opt/py273/bin/python
virtualenv = /opt/py273
# set the http port
http = :8000
# change to django project directory
chdir = /var/www/py/progress_recorder/progress_recorder
# load django
module = django_wsgi:application
env = DJANGO_SETTINGS_MODULE=settings
7.c) Launch uwsgi
/opt/py273/bin/uwsgi --ini django.ini
7.d) In your browser, test uwsgi is working with Django
-Go to:
http://localhost:8000/progress_recorder
-you should see a Django template view displayed via uWSGI itself acting as a web server.
7.e) Stop uwsgi
Ctrl + c
Check this has worked with:
ps aux | grep uwsgi
-if you see any uwsgi processes still running, for each process ID:
kill -9 [processid]
8) Get nginx working with uwsgi
8.a) Check the nginx user's user name
head /etc/nginx/nginx.conf
-you should see in there:
user www-data;
8.b) Make a project configuration file for nginx
(uwsgi support is built into nginx)
cd /etc/nginx/sites-enabled
vim progress_recorder.conf
server {
listen 80;
server_name localhost;
location / {
root /var/www/py/progress_recorder/progress_recorder;
uwsgi_pass unix:///tmp/progress_recorder.sock;
uwsgi_modifier1 30;
include uwsgi_params;
#autoindex on;
}
}
-Save the project config file.
8.c) Alter the Django project's uwsgi ini file to use a socket
-remove the http setting, and insert a new line instead.
[uwsgi]
pythonpath = /opt/py273/bin/python
virtualenv = /opt/py273
# New
socket = /tmp/progress_recorder.sock
# change to django project directory
chdir = /var/www/py/progress_recorder/progress_recorder
# load django
module = django_wsgi:application
env = DJANGO_SETTINGS_MODULE=settings
8.d) Start uwsgi as the nginx user, and check that socket is picked up
sudo su - www-data
(Enter your password)
/bin/bash (to get a decent bash shell)
cd [project_directory], i.e.:
cd /var/www/py/progress_recorder/progress_recorder
/opt/py273/bin/uwsgi --ini django.ini
-you should see something like this in the output:
wsgi socket 0 bound to UNIX address /tmp/progress_recorder.sock fd 3
8.e) Reload nginx with the new configurations
service nginx reload
8.f) Check that nginx and uwsgi are playing nicely
-In your browser, go to:
http://localhost/progress_recorder
-you should see the Django template view displayed via both nginx and uWSGI.
Congratulations!
Subscribe to:
Posts (Atom)