ColdFusion internally represents structure keys in uppercase when the keys are created using dot notation. Dot notation is typically how programmers write their code and is considered to be best practice when working with structures. To the same effect, ColdFusion’s implementation of serializeJSON() serializes your object with respect to case; therefore, we might end up with something like this:

<cfset struct = {} />
<cfset struct.foo = "bar" />
 
<cfdump var="#serializeJSON(struct)#" />
 
<!---
	Result:  {"FOO":"bar"}
--->

Bummer. You’d probably expect “FOO” to be lowercase since that’s how you wrote it, but unfortunately it’s uppercase according to CF. Serializing a query forces uppercase JSON keys as well because queries act and function similarly to structures. This is rather annoying seeing how JavaScript is case sensitive, and serializeJSON is the way to push data from CF to JS. jQuery UI’s Autocomplete is a fine example of a widget that was not designed with case in mind; your JSON is expected to be returned with either a label or value key or both… in lowercase.

If you’re trying to serialize a query, there isn’t much you can do other than converting it to a structure and avoiding dot notation. Use bracket syntax, structInsert(), or create your structure literally:

<!--- 
	three different ways to add values to a structure 
	while forcing lowercase keys 
--->
<cfset struct1 = {} />
<cfset struct1["foo"] = "bar" />
 
<cfset struct2 = { "foo" = "bar" } />
 
<cfset struct3 = {} />
<cfset structInsert(struct3, "foo", "bar") />
 
<cfdump var="#serializeJSON(struct1)#" /> <!--- Result:  {"foo":"bar"} --->
<cfdump var="#serializeJSON(struct2)#" /> <!--- Result:  {"foo":"bar"} --->
<cfdump var="#serializeJSON(struct3)#" /> <!--- Result:  {"foo":"bar"} --->

If you’re using a remote URL for your source parameter with the jQuery UI Autocomplete widget, most likely you’re using a query to retrieve matches from a database. Converting a CF query to JSON has it’s own set of problems because you’re stuck with extraneous data you probably don’t want: “COLUMNS”, “DATA”, and “ROWCOUNT” depending on how you serialize it. Either way, you’ll never be able to pass a serializeJSON(query) directly to the widget, so let’s kill two birds with one stone by converting it to a structure and forcing the keys to a lowercase string:

<!--- get the matches --->
<cfquery name="q" datasource="#request.ds#">
select product_id as value, product_title as label
from products
where product_title like <cfqueryparam value="%#url.term#%" />
</cfquery>
 
<!--- where we'll hold the data to convert to JSON --->
<cfset data = [] />
 
<cfoutput query="q">
 
	<!--- create a new structure on each iteration, forcing lowercase keys --->
	<cfset obj = {
		"label" = label,
		"value" = value
	} />
 
	<!--- push this match onto the array --->
	<cfset arrayappend(data, obj) />
</cfoutput>
 
<!--- serialize the new structure --->
<cfoutput>#serializeJSON(data)#</cfoutput>

Easy peasy, nice and hackish.

This one can be filed under both “total impractical” and “somewhat useful”. For those who haven’t heard of this trick before, base64 encoding enables you to embed binary image data directly into your HTML or CSS source. Therefore, instead of the browser requesting two files from the server – the CSS file and the image file – only the CSS file need be requested. There are a number of drawbacks/pros/cons to this approach: IE doesn’t support it, file size limitations, larger CSS files, inability to separately cache CSS vs. images, etc. etc., but if you feel the need to use FamFamFam Silk icon’s in base64, look no further.

• The CSS file. It’s 2.1mb, so be careful.
• Examples

I use icons very liberally on all my projects; they’re classy and aid in usability. I typically start with a “.icon” class that configures the correct padding/background details for the element, followed by a “.icon-*” class, which sets the background-image property to that of the desired icon. For example, this CSS:

.icon { background-position:left; background-repeat:no-repeat; padding:3px 0 3px 19px }
.icon-pdf { background-image: url(/images/icons/pdf.png) }

Applied to this anchor element:

<a href="#" class="icon icon-pdf">Download PDF</a>

Produces:  Download PDF

Can you dig it? The same .icon-pdf class in base64 looks like this:

.icon-pdf { background-image:url(data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAABAAAAAQCAYAAAAf8/9hAAAABGdBTUEAAK/INwWK6QAAABl0RVh0U29mdHdhcmUAQWRvYmUgSW1hZ2VSZWFkeXHJZTwAAAHhSURBVDjLjZPLSxtRFIfVZRdWi0oFBf+BrhRx5dKVYKG4tLhRqlgXPmIVJQiC60JCCZYqFHQh7rrQlUK7aVUUfCBRG5RkJpNkkswrM5NEf73n6gxpHujAB/fOvefjnHM5VQCqCPa1MNoZnU/Qxqhx4woE7ZZlpXO53F0+n0c52Dl8Pt/nQkmhoJOCdUWBsvQJ2u4ODMOAwvapVAqSJHGJKIrw+/2uxAmuJgFdMDUVincSxvEBTNOEpmlIp9OIxWJckMlkoOs6AoHAg6RYYNs2kp4RqOvfuIACVFVFPB4vKYn3pFjAykDSOwVta52vqW6nlEQiwTMRBKGygIh9GEDCMwZH6EgoE+qHLMuVBdbfKwjv3yE6Ogjz/PQ/CZVDPSFRRYE4/RHy1y8wry8RGWGSqyC/nM1meX9IQpQV2JKIUH8vrEgYmeAFwuPDCHa9QehtD26HBhCZnYC8ucGzKSsIL8wgsjiH1PYPxL+vQvm5B/3sBMLyIm7GhhCe90BaWykV/Gp+VR9oqPVe9vfBTsruM1HtBKVPmFIUNusBrV3B4ev6bsbyXlPdkbr/u+StHUkxruBPY+0KY8f38oWX/byvNAdluHNLeOxDB+uyQQfPCWZ3NT69BYJWkjxjnB1o9Fv/ASQ5s+ABz8i2AAAAAElFTkSuQmCC) }

With this strategy in mind, I took each Silk icon and created a .icon-* class, where * is the name of the icon (underscores replaced with a hyphen). Each icon has a background-image property of the base64 encoded image.

The Code

The code to convert a directory of icons into base64 representation is very straightforward and revolves around ColdFusion’s fileReadBinary() and toBase64():

<cffunction name="base64CSS" output="false" returntype="void">
	<cfargument name="imgdir" required="true" type="string" hint="the directory of images to base64 convert" />
	<cfargument name="destination" required="true" type="string" hint="the path and name to dump the css file" />
	<cfargument name="examples" required="false" type="string" hint="the path and name to dump the example html" />
	<cfset var images = "" />
	<cfset var icon = {} />
 
	<!--- read the image directory --->
	<cfdirectory name="images" directory="#arguments.imgdir#" />
 
	<!--- write our CSS and example files --->
	<cfset filewrite(expandpath(arguments.destination), "") />
	<cfif structkeyexists(arguments, "examples")>
		<cfset filewrite(expandpath(arguments.examples), "") />
	</cfif>
 
	<!--- loop through each image --->
	<cfloop query="images">
 
		<!--- read the image and convert it to base64 --->
		<cfset icon.data = toBase64(fileReadBinary(directory &'/'& name)) />
 
		<!--- famfamfam icons separate words with an underscore.  hyphens look better in CSS --->
		<cfset icon.name = replace(replace(name, "_", "-", "all"), '.png', '') />
 
		<!--- create the actual CSS and HTML --->
		<cfset icon.css = ".icon-" &icon.name& " { background-image:url(data:image/png;base64," &icon.data& ") }" />
		<cfset icon.html = '<div><span class="icon icon-'&icon.name&'">' & icon.name & '</span>' />
 
		<!--- append to the css file --->
		<cffile action="append" file="#expandpath(arguments.destination)#" output="#icon.css#" />
 
		<!--- append to the examples file --->
		<cfif structkeyexists(arguments, "examples")>
			<cffile action="append" file="#expandpath(arguments.examples)#" output="#icon.html#" />
		</cfif>
	</cfset></cfloop>
</cffunction>
 
<cfset base64CSS("/home/ehynds/Assets/icons/silk/", "icons.css", "examples.htm") />

I started by creating a “base64CSS” function which accepts three arguments: a path to a directory of images, the CSS file to generate, and the examples HTML file to generate. cfdirectory reads the image directory, the two CSS/example files are created, and then I loop through each image in the directory. Each image is read as binary data and converted to base64, given a name (separated with hyphens instead of underscores), and the CSS and HTML is generated. Finally, the CSS is appended to the CSS file and the HTML to the examples file.

Simple, right? Does anyone else use some kind of icon class? I’d like to see other approaches!

Related posts:

1. CFIMAGE resize after CFFILE upload

Web developers typically choose one of two ways to paginate data:

1. Select every row in one query and use server-side programming to handle the pagination and output.
2. Use the LIMIT clause in the query to only select the amount of rows necessary for that page of results.

Which ever you choose, each can have a huge performance implication particularly when the data set is in the thousands of rows, or if the query to select the data is not optimized.

The performance hit in option #1 is obvious: you’re selecting ALL the rows when only a fraction of them are going to be displayed to the user at one time. Depending on the situation and if the query can be cached, I suppose this is a fine option, but in my opinion, pagination is a job for your database layer.

In option #2 the query is only going to return XX number of rows out of the entire recordset. We still need to know how many total rows exist in order to figure out how many pages there will be, and to display other meta data to users (showing records 1-100 of 4000, for example). Therefore, a second query still needs to be issued using a COUNT() statement, or the exact same query again without the LIMIT clause… not very efficient.

Although it doesn’t seem well known in the CF community, this scenario is the perfect fit for MySQL’s SQL_CALC_FOUND_ROWS and FOUND_ROWS(). Basically, this statement and function used together gives us the ability to retrieve the total number of rows that would have been returned had the LIMIT statement not been used. Two queries still have to be issued, but this time without the performance hit. Assuming our start_record and records_per_page variables are already set, it works like this:

<cftransaction>
	<cfquery name="getData" datasource="mydatasource">
	SELECT SQL_CALC_FOUND_ROWS field1, field2, field3
	FROM ...
	WHERE ...
	LIMIT #start_record#, #records_per_page#
	</cfquery>
 
	<cfquery name="getDataCount" datasource="mydatasource">
	SELECT FOUND_ROWS() AS totalrecords
	</cfquery>
</cftransaction>

The query getData retrieves only the rows we’re going to output, whereas query getDataCount retrieves the number of rows getData would have returned without the LIMIT restriction. Note that you can issue a SELECT FOUND_ROWS() statement at any time whether you’re paginating or not, and it’ll return the recordcount for whatever the last SELECT was. Therefore, in the example above, I’m wrapping both queries in a cftransaction tag so the FOUND_ROWS() will always reference the getData query. If another query is issued at the same time by another user of your website, selecting FOUND_ROWS() will return the recordcount of that query, not from getData. Here’s a full example of how to build the pagination variables (untested!):

<!--- the current page we're on --->
<cfparam name="url.page" default="1">
 
<!--- number of records to display per page / retrieve in the query --->
<cfset records_per_page = 45>
 
<!--- row to start on during our DB retrieve --->
<cfset start_record = url.page * records_per_page>
 
<!--- get our data --->
<cftransaction>
	<cfquery name="getData" datasource="mydatasource">
	SELECT SQL_CALC_FOUND_ROWS field1, field2, field3
	FROM ...
	WHERE ...
	LIMIT #start_record#, #records_per_page#
	</cfquery>
 
	<!--- get total number of rows --->
	<cfquery name="getDataCount" datasource="mydatasource">
	SELECT FOUND_ROWS() AS totalrecords
	</cfquery>
</cftransaction>
 
<!--- total number of pages --->
<cfset total_pages = ceiling(getDataCount.totalrecords / records_per_page)>

In future versions of ColdFusion it would be nice to see some kind of implementation in the “result” attribute of CFQUERY. If your result variable is “r”, for example, it would be cool to be able to reference #r.found_rows#, much like you can with #r.generated_key# during an INSERT. I’m sure MSSQL and other DBMS’s have something similar, but I’ve only had to use this with MySQL. Happy paginating!

Related posts:

1. Large loops = out of memory?

I have a script that loops over a 6MB text file and takes each line to do a large amount of processing: creating structs, lots of queries, calling other CFCs, etc. The problem is this loop will eventually take up 100% of its alloted memory and eventually timeout with an “out of memory” error. I cannot figure out if it is something I am doing wrong or if it is the nature of CF. From what I’ve read on the web, CF is notoriously poor at garbage collection and will not flush its memory until after the request is complete, which leads me to believe there is nothing I can do.

I have looked over the code with others and can 100% confirm that all variables in a function are var scoped, each structure is cleared at the head of the loop, and there are no open loops or areas where memory would obviously be leaking from.

I can also confirm with the following code (found here) that with each 1000 iterations through the loop, 20MB of memory is being used, which eventually builds up until it’s maxed out.

<cfset runtime = CreateObject("java","java.lang.Runtime").getRuntime()>
<cfset memoryUsed = (runtime.totalMemory() - runtime.freeMemory()) / 1024 / 1024>

It was my understanding that with each loop iteration, recreating the structures that get built within the loop clear them from memory, but this apparently isn’t the case. I have also tried tapping into Java to manually force garbage collection, which also doesn’t do a thing. My only solution at the moment is to increase the Java heap size to 2500MB – enough for the damn thing to run – but once it has finished and other requests are made, memory is never flushed. I then force kill CF and restart it. This is fine for my development box, but is not going to fly in production, and this script will probably be run a couple times a month.

So, aside from posting ~500 lines of code across a handful of files, any quick ideas I can investigate? I’m running CF8 on Ubuntu 9.04.

Related posts:

1. Paginating large datasets with MySQL

I ran into an issue today while trying to resize an image with CFIMAGE that was just uploaded with CFFILE. Apparently there are (were) a number of bugs when these two tags are used in succession. In my case, the image would be uploaded fine, and then magically be deleted by the CFIMAGE tag, at which point an error would be thrown because the file could not be found.

Code:

<!--- upload image --->
<cffile action="upload" 
	destination="#expandpath("/images/sales/")#"
	filefield="image"
	nameconflict="makeunique">
 
<!--- resize it --->
<cfimage action="resize" 
	width="640" 
	height="480" 
	source="#expandpath("/images/sales/#file.serverfile#")#"
	destination="#expandpath("/images/sales/#file.serverfile#")#"
	overwrite="yes">

Turns out there is a simple (hot)fix for this: http://kb2.adobe.com/cps/403/kb403411.html

Related posts:

1. Computer Janitor corrupting AIR installs on Ubuntu 8.04
2. FamFamFam Silk Icon Set in Base64 Encoding