Debugging Jenkins

  Sometimes Jenkins does go crazy. Luckily, there is a nice tool to understand what's happening - Script Console.

   It's available on https://${jenkins}/script and allows you to execute Groovy scripts on running Jenkins. Here is an example of script that I used to debug issue with one plugin, just to give a sense of how far you can go:

	println(Jenkins.XSTREAM.getConverterLookup().lookupConverterForType(org.jenkinsci.plugins.xvfb.XvfbEnvironmentConverter.class))
	def lookup = Jenkins.XSTREAM.getConverterLookup()
	def field = lookup.getClass().getDeclaredFields()[0]
	field.setAccessible(true);
	println(field)
	lookup = field.get(lookup)
	println(lookup)
	println(lookup.getClass().getPackage().getImplementationVersion())
	field = lookup.getClass().getDeclaredField('typeToConverterMap');
	field.setAccessible(true);
	println(field.get(lookup)[null]);
	lookup.flushCache()
	Jenkins.XSTREAM.registerConverter(new org.jenkinsci.plugins.xvfb.XvfbEnvironmentConverter(), 10)
	println(Jenkins.XSTREAM.getConverterLookup().lookupConverterForType(org.jenkinsci.plugins.xvfb.XvfbEnvironmentConverter.class))

view raw sample.groovy hosted with ❤ by GitHub

Jackson @Unwrapped with type information

If you are reading this, then it is highly possible that you know that currently Jackson @Unwrapped does not play nice with type information.

 Actually, in my case the issue was only with serialization. Jackson did recognize my type information, embedded as @type property while deserializing. On serialization it produced something like {:{"a":1}}. But I wanted {"@type":"myType", "a":1}.

Jackson has this issue registered in issue tracker, but it looks like correct fix is too complex. But I still want inheritance with unwrapping :) Here is a snippet that fixes issue.

NB! This code snippet fixes issue, but it's global effect is not confirmed :D

	public class UwrappedWithTypeInformation {
	public static void main(String... args) {
	new ObjectMapper().registerModule(new SimpleModule() {
	public void setupModule(SetupContext context) {
	super.setupModule(context);
	context.addBeanSerializerModifier(new BeanSerializerModifier() {
	public JsonSerializer<?> modifySerializer(
	SerializationConfig config,
	BeanDescription beanDesc,
	JsonSerializer<?> serializer) {
	if (serializer instanceof BeanSerializer) {
	return new BeanSerializer((BeanSerializerBase) serializer) {
	@Override
	public JsonSerializer<Object> unwrappingSerializer(NameTransformer unwrapper) {
	return new UnwrappingBeanSerializerWithTypeInformation(this, unwrapper);
	}
	};
	}
	return serializer;
	}
	});
	}
	});
	}
	public static class UnwrappingBeanSerializerWithTypeInformation extends UnwrappingBeanSerializer {
	public UnwrappingBeanSerializerWithTypeInformation(BeanSerializerBase src, NameTransformer transformer) {
	super(src, transformer);
	}
	@Override
	public void serializeWithType(Object bean, JsonGenerator jgen, SerializerProvider provider, TypeSerializer typeSer) throws IOException {
	if (_objectIdWriter != null) {
	_serializeWithObjectId(bean, jgen, provider, typeSer);
	return;
	}
	jgen.writeStringField(typeSer.getPropertyName(), typeSer.getTypeIdResolver().idFromValue(bean));
	if (_propertyFilterId != null) {
	serializeFieldsFiltered(bean, jgen, provider);
	} else {
	serializeFields(bean, jgen, provider);
	}
	}
	}
	}

view raw UwrappedWithTypeInformation.java hosted with ❤ by GitHub

Why you should autowire constructor instead of field or property

This is quite hot topic in blogs, just wanted to put short and clear list of arguments in favor of constructor injection.

• Autowired field or property can not be used in constructor, need special callback method for that
• Can not use final on autowired field or property
• No separation of concerns. Code with autowired fields becomes Spring specific. Others don't support it.
• Eliminating few lines of boilerplate required for constructor injection will not save you much time.
• With field or property injection code becomes unreadable for those who are not familiar with IOC.
• Field and property injection defeats purpose of constructor.
• If constructor needs too many arguments, it's a sign for refactoring.
• Most developers don't know all details of autowiring properties and fields and it's wrong to say that code becomes more readable.
• Property and field injection hides dependencies instead of making them explicit.

Happy autowiring :)

Restricting system resource access in JUnit tests - SecurityManager

JUnit is not the best framework for integration tests, but it definitely is the most popular one. Today I'm going to describe one weird integration test, where you have to make sure that code under test does not write anything to file system o_O.

Why? Well, for example you application runs in cloud, where quite often file system is not available. Probably you should not go paranoid about making such test for your own code base, but external libraries might give you quite unpleasant surprise. They will work fine on your machine and will crash as soon as you deploy them to sand boxed environment.

Which library can require such test? Basically, any library can attempt to store temp file, for example. Offcourse, good libraries won't do that, but sometimes you don't have any choice.

Java Security has lot's of interesting stuff that many Java developers don't use in their everyday job. One of them is SecurityManager. SecurityManager is responsible for controlling system resource access. One example of SecurityManager in action is that Java applets can't access files on client  machine. Looks like we need something similar for our integration test.

	import java.security.Permission;
	import org.junit.rules.ExternalResource;
	public class RestrictedFileWritingRule extends ExternalResource {
	@Override
	protected void before() throws Throwable {
	super.before();
	System.setSecurityManager(new SecurityManager() {
	@Override
	public void checkWrite(String fd) {
	throw new SecurityException();
	}
	@Override
	public void checkPermission(Permission perm) {
	return;
	}
	});
	}
	@Override
	protected void after() {
	System.setSecurityManager(null); // or save and restore original
	super.after();
	}
	}

view raw RestrictedFileWritingRule.java hosted with ❤ by GitHub

	import org.junit.Rule;
	import org.junit.Test;
	public class SampleTest {
	@Rule public RestrictedFileWritingRule rule = new RestrictedFileWritingRule();
	@Test(expected = SecurityException.class)
	public void writingToFileIsNotAllowed() throws IOException {
	File.createTempFile("asf", "sdfg");
	}
	}

view raw SampleTest.java hosted with ❤ by GitHub

This solution uses JUnit rules for better portability across tests. General idea is that you configure SecurityManager before test run with custom SecurityManager. After test you remove that SecurityManager. Custom implementation throws SecurityException when  code tries to write to file.

Besides checking file writes, SecurityManager can check many other system resources. Network connections for example. 

Some code just kills your day...

	public void calculateFileSizeAndDigest(OutputStream os)
	throws DigiDocException
	{
	if(m_logger.isDebugEnabled()){
	m_logger.debug("calculateFileSizeAndDigest(" + getId() + ") body: " +
	((m_body != null) ? "OK" : "NULL") + " base64: " + m_bodyIsBase64 +
	" DF cache: " + ((m_fDfCache != null) ? m_fDfCache.getAbsolutePath() : "NULL"));
	}
	if(m_contentType.equals(CONTENT_BINARY)) {
	byte[] digest = null;
	try {
	MessageDigest sha = MessageDigest.getInstance("SHA-256");
	String longFileName = m_fileName;
	m_fileName = new File(m_fileName).getName();
	FileInputStream fis = new FileInputStream(longFileName);
	byte[] data = new byte[4096];
	int nRead = 0;
	long lSize = 0;
	while((nRead = fis.read(data)) > 0) {
	sha.update(data, 0, nRead);
	lSize += nRead;
	}
	digest = sha.digest();
	setSize(lSize);
	} catch(Exception ex) {
	m_logger.error("Error calculating bdoc digest: " + ex);
	}
	setDigest(digest);
	if(m_logger.isDebugEnabled())
	m_logger.debug("DataFile: \'" + getId() + "\' length: " +
	getSize() + " digest: " + Base64Util.encode(digest));
	return;
	}
	MessageDigest sha = null;
	boolean bUse64ByteLines = true;
	String use64Flag = ConfigManager.instance().getProperty("DATAFILE_USE_64BYTE_LINES");
	if(use64Flag != null && use64Flag.equalsIgnoreCase("FALSE"))
	bUse64ByteLines = false;
	try {
	sha = MessageDigest.getInstance("SHA-1"); // TODO: fix digest type
	// if DataFile's digest has already been initialized
	// and body in memory, e.g. has been read from digidoc
	// then write directly to output stream and don't calculate again
	if(m_origDigestValue != null && m_body != null && os != null) {
	os.write(xmlHeader());
	if(m_logger.isDebugEnabled())
	m_logger.debug("write df header1: " + xmlHeader());
	os.write(m_body);
	os.write(xmlTrailer());
	return;
	}
	String longFileName = m_fileName;
	File fIn = new File(m_fileName);
	FileInputStream fis = null;
	m_fileName = fIn.getName();
	if(fIn.canRead()) {
	fis = new FileInputStream(longFileName);
	if(m_logger.isDebugEnabled())
	m_logger.debug("Read file: " + longFileName);
	}
	if(m_fDfCache != null) {
	fis = new FileInputStream(m_fDfCache);
	if(m_logger.isDebugEnabled())
	m_logger.debug("Read cache: " + m_fDfCache);
	}
	byte[] tmp1=null,tmp2=null,tmp3=null;
	ByteArrayOutputStream sbDig = new ByteArrayOutputStream();
	sbDig.write(xmlHeader());
	// add trailer and canonicalize
	tmp3 = xmlTrailer();
	sbDig.write(tmp3);
	tmp1 = canonicalizeXml(sbDig.toByteArray());
	// now remove the end tag again and calculate digest of the start tag only
	tmp2 = new byte[tmp1.length - tmp3.length];
	System.arraycopy(tmp1, 0, tmp2, 0, tmp2.length);
	sha.update(tmp2);
	if(os != null) {
	os.write(xmlHeader());
	}
	// reset the collecting buffer and other temp buffers
	sbDig = new ByteArrayOutputStream();
	tmp1 = tmp2 = tmp3 = null;
	// content must be read from file
	if(m_body == null) {
	byte[] buf = new byte[block_size];
	byte[] b64leftover = null;
	int fRead = 0, b64left = 0;
	ByteArrayOutputStream content = null;
	if(m_contentType.equals(CONTENT_EMBEDDED_BASE64)) {
	// optimization for 64 char base64 lines
	// convert to base64 online at a time to conserve memory
	// VS: DF temp file base64 decoding fix
	if(m_fDfCache == null) {
	if(bUse64ByteLines)
	b64leftover = new byte[65];
	else
	content = new ByteArrayOutputStream();
	}
	}
	while((fRead = fis.read(buf)) > 0 || b64left > 0) { // read input file
	if(m_logger.isDebugEnabled())
	m_logger.debug("read: " + fRead + " bytes of input data");
	if(m_contentType.equals(CONTENT_EMBEDDED_BASE64)) {
	// VS: DF temp file base64 decoding fix
	if(m_fDfCache != null) {
	if(os != null)
	os.write(buf, 0, fRead);
	sha.update(buf, 0, fRead);
	} else {
	if(bUse64ByteLines) { // 1 line base64 optimization
	b64left = calculateAndWriteBase64Block(os, sha, b64leftover,
	b64left, buf, fRead, fRead < block_size);
	} else { // no optimization
	content.write(buf, 0, fRead);
	}
	}
	} else {
	if(fRead < buf.length) {
	tmp2= new byte[fRead];
	System.arraycopy(buf, 0, tmp2, 0, fRead);
	tmp1 = ConvertUtils.data2utf8(tmp2, m_codepage);
	}
	else
	tmp1 = ConvertUtils.data2utf8(buf, m_codepage);
	sbDig.write(tmp1);
	}
	if(m_logger.isDebugEnabled())
	m_logger.debug("End using block: " + fRead + " in: " + ((fis != null) ? fis.available() : 0));
	} // end reading input file
	if(m_contentType.equals(CONTENT_EMBEDDED_BASE64)) {
	// VS: DF temp file base64 decoding fix
	if(!bUse64ByteLines && m_fDfCache == null)
	sbDig.write(Base64Util.encode(content.toByteArray(), 0).getBytes());
	content = null;
	}
	if(m_logger.isDebugEnabled())
	m_logger.debug("End reading content");
	} else { // content allready in memeory
	if(m_body != null) {
	if(bUse64ByteLines && m_contentType.equals(CONTENT_EMBEDDED_BASE64) && !m_bodyIsBase64) {
	calculateAndWriteBase64Block(os, sha, null, 0, m_body, m_body.length, true);
	m_body = Base64Util.encode(m_body).getBytes();
	} else {
	if(m_contentType.equals(CONTENT_EMBEDDED_BASE64) && !m_bodyIsBase64)
	tmp1 = Base64Util.encode(m_body).getBytes();
	else if(m_contentType.equals(CONTENT_EMBEDDED_BASE64) && m_bodyIsBase64)
	tmp1 = ConvertUtils.data2utf8(m_body, m_codepage);
	else
	tmp1 = ConvertUtils.data2utf8(m_body, m_codepage);
	sbDig.write(tmp1);
	}
	}
	}
	tmp1 = null;
	if(fis != null)
	fis.close();
	// don't need to canonicalize base64 content !
	if(m_contentType.equals(CONTENT_EMBEDDED_BASE64)) {
	// VS: DF temp file base64 decoding fix
	if(!bUse64ByteLines && m_fDfCache == null) {
	tmp2 = sbDig.toByteArray();
	if(tmp2 != null && tmp2.length > 0) {
	sha.update(tmp2);
	if(os != null)
	os.write(tmp2);
	}
	}
	} else {
	// canonicalize body
	tmp2 = sbDig.toByteArray();
	if(tmp2 != null && tmp2.length > 0) {
	//System.out.println("Body: \"" + tmp2 + "\"");
	if(tmp2[0] == '<')
	tmp2 = canonicalizeXml(tmp2);
	if(tmp2 != null && tmp2.length > 0) {
	sha.update(tmp2); // crash
	if(os != null)
	os.write(tmp2);
	}
	}
	}
	tmp2 = null;
	sbDig = null;
	// trailer
	tmp1 = xmlTrailer();
	sha.update(tmp1);
	if(os != null)
	os.write(tmp1);
	// now calculate the digest
	byte[] digest = sha.digest();
	setDigest(digest);
	if(m_logger.isDebugEnabled())
	m_logger.debug("DataFile: \'" + getId() + "\' length: " +
	getSize() + " digest: " + Base64Util.encode(digest));
	m_fileName = longFileName;
	} catch(Exception ex) {
	DigiDocException.handleException(ex, DigiDocException.ERR_READ_FILE);
	}
	}

view raw gistfile1.java hosted with ❤ by GitHub