The Development of E-Portfolio Evaluation Criteria and Application to the Blackboard LMS E-Portfolio

Transcription

1 International Journal of Virtual and Personal Learning Environments, 3(1), 19-36, January-March The Development of E-Portfolio Evaluation Criteria and Application to the Blackboard LMS E-Portfolio Gary F. McKenna, University of the West of Scotland, UK Mark H. Stansfield, University of the West of Scotland, UK ABSTRACT The purpose of this paper is to develop e-portfolio evaluation criteria which will be used to review the Blackboard LMS e-portfolio being used at one Higher Education (HE) institution in the UK as evaluation criteria for reviewing e-portfolio provision does not exist in the literature. The approach taken was to initiate a wide literature search which involved reviewing over 600 articles by their abstract dating from 1995 to The findings show that little has been written about the development of e-portfolio effective practice frameworks. Therefore e-learning effective practice frameworks were used as a basis from which to design and develop an e-portfolio evaluation framework and then apply it to the university case which uses a Blackboard e-portfolio to support Personal Development Plans. The research provides a starting-point for further research into the development of robust e-portfolio evaluation models and frameworks. Keywords: Blackboard, E-Learning, E-Portfolio, Effective Practice, Frameworks, Guidelines, Learning Management System (LMS), Personal Development Plans, Web INTRODUCTION Not all improvements in technology equate to significant advances in educational outcomes as was the case with the introduction of electronic learning (e-learning) in relation to the implementation of educational policies and effective practice (Weaver, 2002). The institutional infrastructure of tertiary educational establishments provide the support needed for DOI: /jvple this new form of learning to function, as well as, access to technologies which can in many cases be beyond the control of the teacher and is dependent on a wider range of services than courses that do not use e-learning (Deepwell, 2007). Further and higher educational institutions are now providing more support for this type of learning as well as more training for teaching staff who are required to engage with new teaching technologies. With the government set to adopt the wide spread implementation of Personal Develop-

2 20 International Journal of Virtual and Personal Learning Environments, 3(1), 19-36, January-March 2012 ment Plans (PDP) using e-portfolios as a means to fulfill the 2010/11 aim of every student in the UK having a PDP e-portfolio as part of the criteria for the completion of a certificate, diploma, or higher degree and in preparation to replace the honours classification (Burgess, 2007, p. 9). It is crucial that educators, institutions, and other stakeholders have evidence in relation to the performance of e-portfolios as a learning medium. This paper will review the available literature of e-portfolio success within education to: (1) identify which aspects of e-portfolios have been evaluated as a learning tool; (2) describe the evaluation strategies used; (3) synthesise the findings; (4) present an example of an e-portfolio project conducted in a higher education institution within the UK; and (5) discuss educational implications and future research directions. The research has highlighted the need for an e-portfolio effective practice framework based on the findings evidenced in the evaluation of an e-portfolio project for a higher educational institution within the UK. As well as evaluating the existing Learning Management System (LMS) from which the e-portfolio was situated the evaluation also involved building a Personal Development Planning (PDP) portfolio using the postgraduate templates provided by the institution, as well as linking the institutional PDP e-portfolio to external Web 2.0 social networking sites such as Bebo, MySpace, and Wix ( This evaluation report identifies a need to develop an evaluation LMS e-portfolio framework based on the findings of other studies. This study will be of interest to educational policy and programme developers, teaching staff, e-learning and e-portfolio developers, faculty heads, Continuing Professional Development (CPD) and Lifelong Learning partners. The evaluation framework developed for this study was adapted from previous studies of e-learning best practices and virtual campus review frameworks. The adapted model was assembled for the purposes of improving monitoring and development processes to enhance the overall quality assurance of e-portfolios. 2. BACKGROUND TO THE CASE STUDY This review evaluated e-learning system criteria at a higher educational institution in the UK, their initiatives for implementing and facilitating Personal Development Plans (PDP), and the Blackboard Learning Management System (LMS). A literature search was conducted to ascertain an evaluation framework that could be used to review the institution s provision for LMS e-portfolio templates. This was because at the time of conducting the literature review there appeared to be a lack of evaluation criteria for assessing e-portfolios. As a result of this it was decided to review the e-learning literature as a basis for identifying e-portfolio assessment criteria that could be incorporated into the development of an e-portfolio evaluation framework. An e-learning framework was adapted from a previous study carried out by Stansfield and Connolly (2009) which has been recently developed and implemented in a multi HE campus environment and deemed suitable for our purposes. 3. HISTORY OF E-PORTFOLIO EVALUATION IN HIGHER EDUCATION Electronic portfolios (e-portfolios) were first reported to assist in learner development in further and higher education in the late 1980s (Schwartz & Bridwell-Bowles, 1987). In relation to the benefits of using portfolios Armitage (1988, p. 16) affirmed that digital-portfolios can help students learn more efficiently due to increasing levels of motivation and facilitating the publication of their work, which in turn has led to higher levels of self-confidence. Other supporters of e-portfolios (Kimball, 2005; Hatzipanagos & Lygo-Baker, 2008) claim that these methods improve on traditional paper-based methods of reflection due to the learner having: (1) more control over content; (2) attained higher levels of reasoning and self-

3 International Journal of Virtual and Personal Learning Environments, 3(1), 19-36, January-March efficacy; and (3) improved learner cognition in relation to the practice of reflection. Recently interest in learning using e- Portfolios has risen across the sector within tertiary education in the UK driven by national policy and initiatives to improve intellectual growth and personal development (JISC, 2007). Advances in Web 2.0 provide new ways for learners to express themselves online using social networking applications which allow them to document their progress. Educational institutions are using PDP e-portfolios (Strivens, 2007, p. 3) to encourage students to record and plot their personal development over the course of their stay at university and to carry out this practice beyond graduation. Other methods used that are less prescriptive include, but are not restricted to, using social networking web applications such as a blog, wiki, personal website, podcast, videocast, animation, poll, quiz, and RSS feeds (McGee & Diaz, 2007; Richardson, 2008). These can be used to communicate with other like-minded learners with similar interests and can include features to link and share e-portfolio information or to crossreference resources. Zhang et al., (2007) claim that advances in the technological capability of Web 2.0 applications can help to improve and support both the practice of constructivist and reflective learning where the learner attempts to: (1) deconstruct didactic instruction from lectures and classroom tutorials; (2) reconstruct the information and build their own knowledge base; (3) evidence, reflect, and document learned theory to make connections with their own lived experience; and (4) make sense of theory and practice to create new knowledge which is unique to each learner. 4. THE DEVELOPMENT OF AN EVALUATION FRAMEWORK FOR REVIEWING E-PORTFOLIO AND LEARNING MANAGEMENT SYSTEMS In order to develop an evaluative framework the authors reviewed the literature to gain an understanding of the types of models and guidelines that have been successfully implemented in other institutions to evaluate e-portfolio initiatives. As few studies were found the decision was made to look at e-learning evaluative frameworks and guidelines that could be used for synthesising a model for evaluating a blackboard LMS and e-portfolio system. The following two subsections discuss the research method and search criteria used for searching and categorising the frameworks and guidelines found in the literature Method Used for Literature-Based Investigation As part of the initial work carried out on the evaluation of a higher educational institution s PDP e-portfolio project, a literature-based investigation was conducted into previous studies which made use of evaluation frameworks for evaluation purposes. The investigation involved examining papers, reports, and web-based content. From the evaluative frameworks that were reviewed during the literature search the most applicable were Wright (2003) and Britain and Liber (2004). We used literature review methodologies to categorise the abstracts from articles according to relevance and dependability principally in relation to their effectiveness for informing decisions about developing an evaluation framework for use with the Blackboard LMS and PDP e-portfolio learning systems (Strauss & Corbin, 1998; Britten, 2002; Gough, 2007). Articles were selected by their abstracts and the two authors independently applied the criteria to categorise each article as descriptive or evaluative. If an abstract was not available or sufficient information could not be found to make a judgment the article was reviewed. If both authors were in disagreement a third opinion was provided by a colleague. The e-portfolio articles were categorised by evaluation domain. The literature was checked against our criteria for accessing the relevance of the evaluation methods being examined. Where applicable we divided studies into groups where the evaluative

4 22 International Journal of Virtual and Personal Learning Environments, 3(1), 19-36, January-March 2012 features were specific to educational content and good design principles relevant to a Learning Management System (LMS) The Literature Search The literature search identified existing key frameworks of evaluation and the attributes that can be measured within an e-learning environment. The adapted e-learning evaluation framework was then used to identify what can potentially be evaluated in an LMS and e-portfolio system. The process involved searching 11 databases for evaluation criteria which are listed as follows: (1) Emerald; (2) ACM; (3) Springerlink; (4) Science Direct; (5) EEBSCO; (6) Eric; (7) Taylor and Francis; (8) Expanded Academic ASP; (9) ISI Proceedings; (10) Sage; (11) Zetoc. Relevant e-learning evaluation frameworks from journal papers dating from the year 2000 were searched using a mixture of key search terms. The key search terms used consisted of keywords and phrases such as: criteria OR evaluation OR framework OR guidelines ] AND [ e-learning OR elearning OR online courses OR digital learning OR online education. The search returned over 600 articles out of which 14 were found to be appropriate for our search criteria, that is e-learning evaluation guidelines and evaluation frameworks for e-learning. The literature search yielded results which we have used to inform our own criteria for an evaluation framework for evaluating a Blackboards LMS and e-portfolio system Previous Studies that Informed our Framework From the literature search 10 areas were identified as being important for measuring the effectiveness of an LMS e-portfolio system. The 10 areas recognised as suiting our needs are: (1) the general information given to learners prior to commencement of a course (Wright, 2003; QAA, 2004); (2) accessibility of course materials (Ball & Sutherland, 2003; CATEA, 2006); (3) course organisation (Wright, 2003); (4) course language (ANSI, 2004); (5) course layout (ANSI, 2004); (6) course goals and objectives (Mager, 1975; Guilbert, 1984; Gallagher & Smith, 1989; Harrison, 1999); (7) course content (Wright, 2003); (8) learning strategies and opportunities for practice and transfer (Hillesheim, 1998; Ragan, 1999; Marshall, 2005); (9) learning resources (Marshall & Mitchell, 2003; Wright, 2003); and (10) assessment (Iahad et al., 2004). The evaluation model we used was created and synthesised from the e- Learning evaluation criteria identified from 14 different sources. In addition, three different evaluation types were identified from the literature in relation to evaluating various stages of the design and development process of an e-learning system (Table 1). For our purposes we used formative evaluation when testing the processes and usability of the curriculum, PDP materials, content system and e-portfolio interface because it allowed us to evaluate the processes and usability at the point of use (Wright, 2006). 5. THE EVALUATION CRITERIA Our evaluation criteria for constructing an evaluation framework consisted of 10 different areas of e-learning criteria and guiding principles identified from both the literature and the discussions held with staff members (Table 2). The e-learning evaluation criteria and guiding principles that make up our evaluation framework are considered important for assess-

5 International Journal of Virtual and Personal Learning Environments, 3(1), 19-36, January-March Table 1. Three evaluation types No. Evaluating Stages of the Design: 3 Types 1. Diagnostic evaluation is used to identify and analyse preconceptions and assumptions prior to use 2. Formative evaluation is used to appraise implementation processes and usability at the point of use 3. Summative evaluation used to assess outcomes in terms of changes in learning and teaching behaviours and achievements (Wright, 2006) ing the effectiveness of the Blackboard learning management system and learning e-portfolio. Within the context of developing evaluation criteria Wright (2003) published a set of criteria for evaluating the quality of online courses which we used as part of our effective practice model to assess the effectiveness of an LMS and e-portfolio system. The evaluation for the PDP e-portfolio templates and Blackboard e- Learning system were divided into 10 sections in accordance with the 10 areas of evaluation we identified form the literature which will be discussed within the following subsections General Information Given to Learners Prior to Commencement of Course A number of criteria have been identified that are important to give to learners prior to the commencement of a course in terms of general information. The criteria relate to information provided at the beginning of a course that will assist the learner to complete the course and to understand its objectives and procedures. Wright (2003) identified a number of detailed criteria in relation to general information that should be made available to learners, examples of which are: Special technology requirements are identified relating to hardware and software specifications needed to take part online. The level of the course is clearly specified for example, using the QAA Scottish Credit and Qualifications Framework, which identifies 12 levels covering access, standard grade, Higher, HND, Degree, M.Sc. and Doctorate. Prerequisites are identified including the level of IT skills required to complete an online course. A brief description of the course includes goals and learning objectives and learning outcomes is provided (QAA, 2004). A list of required and recommended resources including all textbooks, courseware, and online resources necessary to complete the course is made available. If learners must access online databases, instructions are provided for locating and accessing these resources, including password information. If the materials are located in a library, learners are told whether the items are on reserve and, if so, for how long. Guidelines for participating in online discussions, ( netiquette ) are provided, as well as suggestions for handling incoming , attachments, viruses, and filters. Learners are informed about guidelines and grading criteria for assignments as well as their responsibilities as group members of a class. Learners are directed to a source for answers to frequently asked questions pertaining to the LMS and PDP/e-Portfolio. Learners are informed about their right to privacy and the conditions under which their names or online submissions may be shared with others Accessibility of Course Material With reference to the importance of making course materials accessible online by implementing accessibility processes that ensure students can actually access the materials on an equal basis, the CATEA (2006) highlighted the importance of accessibility of course material within an online learning environment. They suggest reviewing course materials within the context of learning barriers so as to anticipate difficulties that learners may face when attempting to access online materials. Other

6 24 International Journal of Virtual and Personal Learning Environments, 3(1), 19-36, January-March 2012 Table 2. Evaluation criteria No. Evaluation Criteria for Curriculum (PDP) Materials, and Content System 1. General Information Given to Learners Prior to Commencement of Course: Prior to the start course learners are provided with general information to assist them in completing the course and in understanding its objectives and procedures. Any special technology requirements are identified such as software and hardware specifications needed to support online learning (Wright, 2003; QAA, 2004). 2. Accessibility of Course Material: Course materials are made accessible online to all learners on an equal basis, in terms of learning barriers, materials should be reviewed to anticipate difficulties that learners might face when attempting to access online materials (Ball & Sutherland, 2003; CATEA, 2006). 3. Course Organisation: The organisation in relation to the layout of materials is arranged appropriately for the subject matter and the intended audience (Wright, 2003). 4. Language: includes spelling, grammar, punctuation, format, reading level, cultural bias, technical terms and screen layout (ANSI, 2004). 5. Layout: Within the context of web design computer screen layout include characteristics such as: margins, white space, and physical readability should follow accepted conventions (ANSI, 2004). 6. Course Goals and Objectives: Outline what learners should expect to learn from a course and where appropriate, what they should expect to learn at the beginning of each module. They specify learning outcomes related to knowledge, skills, competencies, behaviours and attitudes (Mager, 1975; Guilbert, 1984; Gallagher & Smith, 1989; Harrison, 1999). 7. Course Content: Should be designed so as to be assessable to learners form various backgrounds and abilities (Wright, 2003). 8. Learning Strategies and Opportunities for Practice and Transfer: Encapsulates the need to provide learning opportunities for learners of different learning styles. For many learners e-learning technologies and their associated pedagogies can be unfamiliar to them and the range of possibilities can be diverse. It is therefore important to provide information and opportunities for learners so as to familiarise them with what to expect (Hillesheim, 1998; Ragan, 1999; Marshall, 2005). 9. Learning Resources: provide learners with information or preparation in order that they can make meaningful context and effective use of it in their learning. Learning materials are appropriate for the learners and the subject matter and the learning resources are accessible, appropriate, accurate, and related to the course content. Lists of learning resources are divided into required and optional categories. Various learning resources are used to ensure compatibility with learners different interests, abilities, and learning styles. Learning resources reflecting different points of view are provided when appropriate (Marshall & Mitchell, 2003; Wright, 2003). 10. Assessment: activities are feasible, relevant, accurate, and congruent with the objectives, content, and practical applications of the content. There are one or more components of assessment to demonstrate that the necessary learning outcomes have been achieved by the learner. Clear expectations are given with the criteria for assignments. Along with Examples of assignments that meet the criteria. The number of assignments and their due dates are reasonable. Grading policies are clear and explicit (Iahad et al., 2004). factors that should be considered in relation to accessibility are the usability of the course materials in addition to broken web links as well as the rewording of confusing directions. Quality e-learning Web resources should be usable and not just accessible (Ball & Sutherland, 2003). However, there may be instances where complying with accessibility guidelines might conflict with the usability of the web page or require undesirable changes to the web pages appearance Course Organisation A number of evaluative criteria and guiding principles concerning course organisation have been highlighted as essential if learners are to make connections between different parts of the course. Examples include the organisation

7 International Journal of Virtual and Personal Learning Environments, 3(1), 19-36, January-March of units of instruction as well as topics are divided into subunits of instruction and subtopics and organisation of components are consistent across the course. References to other parts of the material are correct including links to other parts of the course and external sources of information are accurate (Wright, 2003; Britain & Liber, 2004) Course Language According to ANSI (2004) within the context of language there are several important factors that need to be considered when designing e-learning materials. These design factors consist of criteria for checking correct spelling, grammar and punctuation of course materials in addition to the reading level being audience and content specific. Other factors involve defining technical terms correctly and consistently throughout the course content as well as avoiding cultural bias. Cultural bias may include assumptions made about attitudes, beliefs, and opinions that relate to a specific culture, therefore course content should be usable in cross-cultural contexts. Within the context of readability the reading level will vary and depend on the content, for instance, technical content will usually have a higher reading level due to the number of multi-syllable words used to describe something technical. Instructions, manuals and non-technical content will tend to have a lower reading level. In relation to consistency technical content should be at a single reading level, as well as test questions and instructions. Technical terms should only be used if they are relevant to the content and should include a definition that is understandable to a novice. An online glossary or hyperlinks to term definitions should be considered. Abbreviations or acronyms are specific to a field their initial use should contain an explanation Course Layout Other important factors highlighted by ANSI (2004) concern guiding principles for designing the layout of the Graphical User Interface (GUI) for an e-learning environment. They suggest the following guidelines: that the layout for computer screens should be easy to follow with reference to how space and colour is used. Bad screen design can cause confusion for learners with regards to navigation. Spaces between paragraphs should be used to increase the readability of scrolled text. The space between content areas on the screen should contain sufficient space to set them apart and increase the readability of the text. The same spacing should be used throughout the product (Britain & Liber, 2004) Course Goals and Objectives According to Mager (1975) course goals and objectives describe a performance that learners should be capable of demonstrating before they can be considered competent. Clearly written course goals and objectives indicate to learners what they should expect to get out of a course with regards to what they will learn from each module and the course as a whole. They should make explicit the learning outcomes that build new knowledge, skills, competencies, behaviours and attitudes (Harrison, 1999). They should be pertinent to the course content and be applicable in the real world where the content may be applied. In addition, each main section of a course should have specified measurable learning objectives. Guilbert (1984) identified six characteristics of course goals and learning objectives which he considers being important for learners if they are to gain new knowledge and are listed as: (1) Relevant; (2) unequivocal; (3) feasible; (4) logical; (5) observable; and (6) measurable. Others that have written about course goals and objectives are Gallagher and Smith (1989), who state that properly constructed education objectives represent relatively specific statements about what students should be able to do following instruction Course Content In relation to evaluative criteria for course content Wright (2003) proposes several guidelines relating to effective practices that should be

8 26 International Journal of Virtual and Personal Learning Environments, 3(1), 19-36, January-March 2012 considered when evaluating course content such as: Learners from different backgrounds and varying abilities should be able to understand the course content as well as carryout instructions for tasks. The content should be directly linked to the learning objects, complete in terms of providing all the content or learning experiences needed to achieve the learning objectives and be appropriate to the learners characteristics (ability and maturity level). The course content should be accurate, relevant, current, broken into small, incremental learning steps, presented in a logical sequence, and related to other material the learners may have studied or experiences they may have had. It should make use of illustrated examples and/or case studies when new information is presented and linked to other sources. Reading assignments are clearly specified and all quoted materials should be cited correctly, in addition to, permission for all copyrighted material used has been obtained Learning Strategies and Opportunities for Practice and Transfer Within the context of providing opportunities for practice and transfer there are five levels of characteristic practices operating at each level of process capability identified by Marshall (2005, p. 37). They are: (1) Initial, the ability of learners to practice and prepare for the use of technologies in online courses is incidental to the provision of materials or systems; (2) planned, opportunities for learners to practice and develop the necessary skills for using e-learning technology use are explicitly identified in the course materials available prior to commencement of the course; (3) defined, there are standards for providing learners with preparation and practice opportunities for all standard technologies which are defined along with templates and examples for teaching staff to use in course materials. (4) managed, compliance with standards for providing learners with preparation and practice opportunities is measured, as is learner satisfaction and preparedness to use technologies; (5) optimised, in relation to measurements of effectiveness of standards for providing learners with sufficient preparation and practice are used to maintain and update the standards and the requirements for courses generally (Hillesheim, 1998; Ragan, 1999; Britain & Liber, 2004; Marshall, 2005). Learners preparedness should be taken into account prior to new technology introduction. In terms of being better prepared to get the most benefit out of e-learning technologies many learners will need to make plans to ensure they have the necessary skills to be successful in taking part in online learning. Ensuring that learners have adequate information before a course commences will help to prevent early withdrawal or at a later stage (Hillesheim, 1998; Ragan, 1999) Learning Resources Writing about process indicators for evaluating learning resources in e-learning Marshall and Mitchell (2003) highlight the following recommendations. Firstly, students are provided with sufficient background information or preparation that they can place any resource within a meaningful context and make effective use of it in their learning. Select and apply pedagogical strategies which are based on current and relevant educational research and which are suited to the particular learning resources. Determine what technologies will be used to support the delivery of course resources and student interaction. Ensure that the requirements of use of any technology are consistent with the student characteristics and that technology is selected for its educational benefits and not just for its own sake (Wright, 2003) Assessment The need to evaluate online assessment processes and the role of feedback in learner-centred e-learning has been identified and highlighted by Iahad et al. (2004). They state several key factors that are important to consider when evaluating online assessment criteria. They are that, the relationships between course learning outcomes and assessment strategies are evident

9 International Journal of Virtual and Personal Learning Environments, 3(1), 19-36, January-March to the learner and between assignments as well as the final course grade being clearly specified. Detailed step-by-step instructions are provided for each component of assessment along with guidelines for submitting assignments. Learners are informed about the criteria that will be used to evaluate their participation in online activities such as discussion groups. A variety of feasible and content-relevant assignments are provided. The assessments are relevant to the learners and the career or profession they may pursue. Learners are able to track and evaluate their own progress and self-tests are similar to the final assessments. If appropriate, learners are informed about the consequences of plagiarism and the failure to properly cite copyrighted material. Learners are told when they can expect to receive feedback from the instructor. Assessment procedures are congruent with the objectives and reflect any priorities that have been established for the objectives (Britain & Liber, 2004). As a more general issue, e-learning materials are generally developed for particular learner populations. With some of the early discussions between the reviewer and the Continuing Academic Personal Development (CAPD) staff, there seemed to be an aim to market the PDP materials together with faculty course content. Having PDP initiatives and differing faculty objectives could prove problematic for staff and students. A more integrated strategy might be needed in terms of a more unified approach by faculties towards personal development, at the moment it seems to be left to the discretion of each faculty, as to how they go about embedding PDP into their own curriculum frameworks. 6. THE PROJECT: E-PORTFOLIO EXEMPLAR After the e-portfolio evaluation criteria was identified from previous e-learning evaluation frameworks identified from the systematic literature search. An e-portfolio exemplar was built from the Blackboard LMS e-portfolio system used at a UK HE institution which was then evaluated with the evaluation framework identified earlier in this study. A report was produced for the (CAPD) staff at the HE institution which highlighted issues found and provided solutions and recommendations about the LMS e-portfolio exemplar. This was used as a pilot to test out the LMS e-portfolio prior to being used by students at the HE for recording their PDP. To achieve this one sample e-portfolio was constructed based on the PG Certificate in Education templates for evidencing PDP which was informed by teacher placement experience of the researcher. External social networking websites were tested to ascertain which sites are best suited for hosting students personal development plans. The results of the review are as follows: The two social websites suited for hosting students PDP that we found were: Bebo and MySpace, although limitations were identified in terms of limitability of not being able to create and customise templates and models for the purposes of setting personal categories and topics of interest. We used Bebo to host an external PDP social website that focused on: The Theories of Learning due to the availability of the uploading features for upload and videos. MySpace was used to host: The Theories of modern education. A website (Wix) was used as an alternative to using social networking websites as some students might not be comfortable with having a web presence on a social website. Another reason for using Wix is that it is a flash driven website which utilises prescripted templates which makes it simpler to create web pages. We used this to host a research page because it provided more control over naming the different categories and topics of the web pages. The problems and issues found with using these social Websites for PDP projects were: (1) RSS feeds could not be accommodated in MySpace; (2) RSS feeds were accessible in Bebo, but were limited to what Bebo provided; and (3) Templates

10 28 International Journal of Virtual and Personal Learning Environments, 3(1), 19-36, January-March 2012 and Modules could not be modified in both Bebo and MySpace in with reference to renaming modules to suit the purposes of the user. As an alternative to using social networking websites the flash driven website was found to be simpler for students to use with little or no experience in web design. This is a viable option for those who do not wish to have their PDP hosted on a social networking website. 7. OVERVIEW OF THE PDP E-PORTFOLIO TEMPLATE SYSTEM 7.1. Building an E-Portfolio within the Blackboard LMS The PDP e-portfolio system was developed to be highly configurable using Blackboard LMS. The design tools allow templates to be created that students can choose from when creating a portfolio. The templates can be selected by undergraduate and postgraduate, examples include: Record of placement activities, my learning in context, critical reflections, year 1 initial skills audit, Trimester goals, feedback and employability related activities which are show in Figure 1. Students can build portfolios based on the context of the course, but teachers also have the ability to alter the generic templates to suit the needs of the course or produce their own framework in template form Template Modifying an Item Using Blackboard The templates can be modified by using the Modify Item selection which presents the student with a set of text tools that can be used to edit text much like a word document. The modifications to text can then be saved and altered or added to at a later date Adding a Web Link to a Template on Blackboard Links to other web sites can be added by using the Add Link feature allowing students to link to both internal and external web pages. In the example, the student has linked their e-portfolio to their Bebo web page by entering the web address in to the Uniformed Resource Locator (URL) under the Specify Link section Sharing the PDP E-Portfolio with Other Students The Blackboard framework allows for the sharing of e-portfolios with other students and members of staff so that comments and feedback can be left about the content of their e- Portfolios. The e-portfolio also facilitates sharing with users who are external to blackboard such as prospective employers, staff from other institutions and careers representatives. This is achieved by the owner of the e-portfolio first adding the external user to their share list. Then an is sent with a set of instructions from the student s blackboard account which invites the external user(s) to access their portfolio via a web link and password Link to Bebo Web 2 Social Networking Site Created on Blackboard Template Links to Bebo, MySpace and Wix were integrated as external links into the LMS e-portfolio system to allow students to share their personal interests with one another. This was done for the following two reasons. First, it was thought that this would allow students to include their activities out with the university into the LMS e-portfolio system providing a link to their outside interests. Second, it an attempt to link the PDP learning experience to student extra curricula activities bringing together all of their interests. This could be used by the students to provide employers with a more comprehensive

11 International Journal of Virtual and Personal Learning Environments, 3(1), 19-36, January-March Figure 1. PDP template selection available for students: PGCert Academic Writing Skills employability profile. To achieve this from the main e-portfolio page a users link was created that links to the Bebo social networking site. Figure 5 shows the Bebo page displayed within the e-portfolio after clicking the link shown in the left user s panel. The Bebo site can be navigated from within the e-portfolio making visiting other websites and retrieving information simple without leaving the e-portfolio Link to MySpace Page Created on Blackboard Template The social web 2 website MySpace has been added and can be accessed from the main e- portfolio users page. Figure 6 shows MySpace web page being displayed from within the e-portfolio user page which can be accessed without navigating away from the e-portfolio. The Myspace web site can be browsed and Figure 2. Modifying text selection: example from postgraduate certificate in teaching

12 30 International Journal of Virtual and Personal Learning Environments, 3(1), 19-36, January-March 2012 Figure 3. Adding a link to other internal or external web pages and web sites information can be downloaded from within the e-portfolio Linking to My Research Page on Wix Created on Blackboard Template The web site creator by WiX was used as a Research Log web site and is an alternative to having an external web presence without using social networking sites (as shown in Figure 7). Some students might prefer to build an external web site instead of a social web 2 web site. Building an external web site with WiX is made simple by using flash templates which the student can customise to suit their own purposes Managing My E-Portfolios As shown in Figure 8 creating a new portfolio and receiving access to other students portfolio(s) is made simple with the Portfolio management tool. From this page a student can keep track of their own e-portfolio(s) and make Figure 4. Sharing an e-portfolio with internal or external users

13 International Journal of Virtual and Personal Learning Environments, 3(1), 19-36, January-March Figure 5. Linking to Bebo Web 2 social networking site modifications or view received portfolios that other students have shared with them Downloading and Saving My E-Portfolio The LMS e-portfolio system allows students to save their entire e-portfolio in a.zip file format (shown in Figure 9) which can be imported onto another learning management System (LMS) if a student decides to move to another institution or if after graduating progresses onto a different course at another institution. Students might also wish to keep a permanent record of their PDP progress or wish to continue with their progress file for professional development purposes when starting a job after graduation. 8. NEED TO REDEFINE THE LMS E-PORTFOLIO EVALUATION CRITERIA LMS e-portfolio evaluation criteria were developed based upon e-learning evaluation frameworks for the purposes of reviewing the suitability of the LMS e-portfolio system used to record students PDP at a higher educational institution within the UK between 2009 and The e-portfolio exemplar was created as an example to demonstrate what can be achieved with a blackboard e-portfolio framework. Although the e-portfolio evaluation framework developed for this study was successfully run, but since its initial implementation the LMS e-portfolio system has undergone changes as a result of the University s Blackboard system being upgraded. This has resulted in certain aspects of the LMS e-portfolio evaluation criteria having to be revised to take into account changes and usability upgrades to the system. Once the Blackboard upgrade has been completed across all of the institution s different campuses the redefined evolution framework can then be applied to the LMS e-portfolio. For educational purposes the study was undertaken to highlight any short comings of the frame work that was being use to implement PDP within a Higher Educational institution within the UK. In order to do this an Evaluation criteria for Curriculum (PDP) Materials, and Content System was developed as a framework from which to review the framework being implemented across four campuses for 2009/2010 (Table 2). Prior evaluative models and frameworks have been developed or e- Learning initiatives but not much has been developed for e-portfolios. More research is needed to provide alternate models and frameworks for educationalists and institutions to use when designing and evaluating the effectiveness of their own e-portfolio initiatives. Using the Blackboard e-portfolio templates a PDP

14 32 International Journal of Virtual and Personal Learning Environments, 3(1), 19-36, January-March 2012 Figure 6. Linking to MySpace Web 2 social networking site e-portfolio exemplar was created with links to external web 2 social networking sites and a web site. The features that can be used when creating a PDP e-portfolio were documented and are shown in Figures 1 through 9. Screen shots were used to illustrate the simplicity and ease of how an e-portfolio can be constructed using the templates provided by each institutions staff responsible for developing PDP guidelines and models of recording progress of personal development. Social networking web 2 web sites were linked to the institutional e-portfolio as an alternative to using the institutional templates and e-portfolio system, as some students had made it clear that they feel more comfortable recording their personal development in an environment that they are already familiar with. 9. FUTURE RESEARCH: THE NEXT STAGE IN THE DEVELOPMENT AND USE OF THE PDP PLATFORMS The next stage of the research in the development and use of the PDP platform for recording achievement will be to develop the evaluative framework and models as well as effective practice models further by testing them out Figure 7. Linking to WiX flash driven web site

15 International Journal of Virtual and Personal Learning Environments, 3(1), 19-36, January-March Figure 8. Management interface for my portfolio(s) and received portfolio(s) across other institutions and their schools or faculties. Schools and departments that wish to participate in the dissemination and piloting of the PDP e-portfolio framework will be sought for the purposes of identifying barriers that may inhibit or hinder the recording of their personal progress. Ways will be sought to overcome these barriers in the form of new solutions or enhancers that will inform educational practice among staff and student. The progress of the research will be documented and the results of the study will be published along with the frameworks and effective practice models. They will be made available for educationalists and institutions to use as guidelines when implementing their own initiatives and evaluations into the effectiveness of PDP e-portfolios. The revised LMS e-portfolio evaluation framework will be used to evaluate the updated version of Blackboard LMS e-portfolio system. At some point in the future the LMS e-portfolio evaluation framework will be applied to other LMSs to compare the different approaches and effectiveness by LMS. An important objective of future research surrounding this study will be to examine how Moodle LMS e-portfolio compares to Blackboard LMS e-portfolio. This would relate to a future research objec- Figure 9. Downloading an e-portfolio and saving to zip file format

16 34 International Journal of Virtual and Personal Learning Environments, 3(1), 19-36, January-March 2012 tive which would be to provide a comparison of the features of the different types of LMS e-portfolio platforms that could be applied to different academic institutions. 10. CONCLUSION Research Objectives The overall objective was to evaluate the LMS e-portfolio system provision at a higher educational institution in the UK, their initiatives for implementing and facilitating Personal Development Plans (PDP), and Blackboard Learning Management System (LMS). From this a literature search was conducted to ascertain an evaluation framework that could be used to review the institution s provision for Electronic Portfolio (e-portfolio) templates. In order to achieve this the paper reviewed the available literature of e-portfolio success within education to: (1) identify which aspects of e-portfolios have been evaluated as a learning tool; (2) describe the evaluation strategies used from e-learning which would be applicable to assessing e-portfolio; (3) present an example of an e-portfolio exemplar which was used to assess the recently developed e-portfolio evaluation framework; and (4) discuss educational implications and future research directions. The adapted model included criteria for the evaluation of curriculum, PDP materials, and the content system. This was grouped into 10 subcategories: (1) General Information Given to Learners Prior to Commencement of Course; (2) Accessibility of Course Material; (3) Course Organisation; (4) Language; (5) Layout; (6) Course Goals and Objectives; (7) Course Content; (8) Learning Strategies and Opportunities for Practice and Transfer; (9) Learning Resources; (10) Assessment The Importance and Benefits of this Paper This study has highlighted the need for evaluative frameworks as well as the need for effective practice models to be developed that can be used to design and evaluate PDP e-portfolio initiatives within Higher Educational institutions. This paper is important because it is the first of several papers that will assert the need to develop evaluative and effective practice models for implementing Personal Development Plans (PDP) using e-portfolios as a means to fulfil the 2010/11 aim of every student in the UK having a PDP/e-portfolio as part of the criteria for the completion of a certificate, diploma, or higher degree and in preparation to replace the honours classification (Burgess, 2007, p. 9). The paper will be of importance to educators, institutions, and other stakeholders seeking evidence in relation to the performance of e-portfolios as a recording and learning medium for PDP. REFERENCES ANSI Standards Committee on Dental Informatics. (2004). Working group educational software systems, ANSI/ADA specification 1001: Guidelines for the design of educational software. Retrieved December 2, 2009, from informatics/edswstd/ Armitage, C. (1998). The benefits of pause for thought (p. 16). The Australian. Britain, S., & Liber, O. (2004). A framework for the pedagogical evaluation of elearning environments: Evaluation of learning management system software part II of LMS evaluation, open source e-learning environment and community platform project. Retrieved October 8, 2009, from org/docman/view.php/7/17/evaluation%20of%20 LMS%20-%20Part%20II.pdf Britten, N., Campbell, R., Pope, C., Donovan, J., Morgan, M., & Pill, R. (2002). Using meta-ethnography to synthesis qualitative research: a worked example. Journal of Health Services Research & Policy, 7(4), doi: /

17 International Journal of Virtual and Personal Learning Environments, 3(1), 19-36, January-March Burgess, R. G. (2007). Beyond the honours degree classification: The Burgess Group final report. Retrieved November 1, 2009, from universitiesuk.ac.uk/publications/documents/burgess_final.pdf Cann, C., Ball, S., & Sutherland, A. (2003). Towards accessible virtual learning environments. TechDis. Retrieved November 1, 2009, from techdis.ac.uk/resources/vle001.html CATEA. (2006). Reviewing and evaluating course materials for accessibility. Retrieved December 18, 2009, from mod2/2_04.php Gallagher, R. E., & Smith, D. U. (1989). Formulation of teaching/learning objectives useful for the development and assessment of lessons, courses, and programs. Journal of Cancer Education, 4(4), doi: / Gough, D. (2007). Weight of evidence: a framework for the appraisal of the quality and relevance of evidence. Applied and Practice-based Research, 22(2), Guilbert, J. J. (1984). How to devise educational objectives. Medical Education, 18(3), doi: /j tb00992.x Harrison, N. (1999). How to design self-directed and distance learning programs: A guide for creators of web-based training, computer-based training, self-study materials. New York, NY: McGraw-Hill. Hatzipanagos, S., & Lygo-Baker, S. (2008, June 26-27). e-portfolios for academic development: a career progression vehicle or a private tool of reflection. In Proceedings of the International Conference on e-learning, Cape Town, South Africa. Hillesheim, G. (1998). Distance learning: Barriers and strategies for students and faculty. The Internet and Higher Education, 1(1), doi: / S (99) Hottos, S. (1993). CD-I designers guide. New York, NY: McGraw-Hill. Iahad, N., Dafoulas, G. A., Kalaizakis, E., & Macaulay, L. A. (2003). Evaluation of online assessment: The role of feedback in learner-centered e-learning. In Proceedings of the 37th Hawaii International Conference on Systems Science. JISC. (2007). e-portfolios: An overview of the portfolio activity area. Retrieved December 2, 2009, from Mager, R. (1975). Preparing instructional objectives (2nd ed.). Belmont, CA: Fearon-Pitman. Mager, R. F. (1984). Preparing instructional objectives: A critical tool in the development of effective instruction. Belmont, CA: Pitman Management and Training. Marshall, S. (2005). NZ e-learning capability determination: Determination of New Zealand tertiary institution e-learning capability: An application of an e-learning maturity model. Report on the e- Learning maturity model evaluation of New Zealand tertiary sector, capability assessment methodology guide. Retrieved November 10, 2009, from MethodologyGuide.pdf Marshall, S., & Mitchell, G. (2003). Potential indicators of e-learning process capability. In Proceedings of the EDUCAUSE Conference, Adelaide, Australia. Retrieved November 10, 2009, from PDF/AUTHOR/ED PDF McGee, P., & Diaz, V. (2007). Wikis and podcasts and blogs! Oh, my! What is a faculty member supposed to do? EDUCAUSE Review, 42(5), Miles, K. (2005). Database e-portfolio systems: A critical appraisal. Computers and Composition, 22(4), doi: /j.compcom Petticrew, M., & Roberts, H. (2006). Systematic reviews in the social sciences: A practical guide. Oxford, UK: Blackwell. doi: / Quality Assurance Agency. (2004). Code of practice for the assurance of academic quality and standards in higher education: Section 2: Collaborative provision and flexible and distributed learning (including e-learning). Mansfield, UK: Quality Assurance Agency for Higher Education. Retrieved December 22, 2009, from Ragan, L. C. (1999). Good teaching is good teaching: An emerging set of guiding principles and practices for the design and development of distance education. Cause/Effect, 22(1). Retrieved November 14, 2009, from cem99/cem9915.html Richardson, W. (2008). Blogs, wikis, podcasts, and other powerful web tools for classrooms. Thousand Oaks, CA: Corwin.