The Integrated Technology Assessment:
A Portfolio-based Capstone Experience
Li-Fang Shih
lshih@excelsior.edu
Jane A. LeClair
jleclair@excelsior.edu
School of Business and Technology
Excelsior College
7 Columbia Circle
Albany, New York 12203 USA
Stuart A. Varden
svarden@pace.edu
Seidenberg School of Computer Science and Information Technology
Pace University
861 Bedford Road
Pleasantville, New York 10570 USA
Abstract
Excelsior College, located in Albany, New York, has been a pioneer institution in distance learning. The Integrated Technology Assessment (ITA) is one particularly unique dimension of student assessment offered by the School of Business and Technology at the College. This portfolio-based assessment method is a capstone experience for the BS Information Technology program. In this capstone course, students are required to demonstrate their ability to integrate knowledge from various technical and general education areas and apply it in a meaningful way. This article describes the development and implementation of the capstone course which requires the information technology students to develop an online portfolio reflecting technical competencies they acquired during their academic studies. The manuscript provides a complete description of the ITA process at Excelsior College. It also provides information regarding the methods used to assess students’ learning and presents a comprehensive review of the development and implementation of this capstone course. Challenges and strategies to cope with the challenges are also discussed at the end of the paper.
Keywords: capstone experience, outcome assessment, non-traditional higher education, portfolio assessment, continuous program improvement
1. INTRODUCTION
Higher education in general struggles to keep up with the changes that are occurring at a lightning speed around us. In order to realign itself with these changes, higher education must be innovative in the areas of openness, connectedness, personalization, participation, as well as the infrastructure of teaching and learning. Openness is the key ingredient that enables innovation and improvement in the quality, accountability, affordability, and accessibility of higher education (Morris, 2005).
With this goal of increased openness in mind, Excelsior College in Albany, New York, was founded in 1971 by the New York State Board of Regents, and was originally known as Regents College. In 1998, it was granted a charter to operate as a private, independent college and changed its name to Excelsior College in 2001. Currently, it has approximately 33,000 enrolled students and is one of the most respected distance learning institutions in higher education.
Recognizing that there are many adult learners who have acquired their knowledge and capabilities through experiences other than formal classroom learning, at the center of the Excelsior College mission and strategic plan is the idea of “What you know is more important than where or how you learned it.” To that end Excelsior College has designed a model that is student centered and responsive to the needs of career-oriented adult learners at a distance.
Student Profile
Excelsior College (EC) has a diverse student population, with 89% of the student body located outside of New York where the College is headquartered. Presented in Table 1 is a brief overview of the student profile of the BS in Information Technology program at EC.
BS in Information Technology
Student Profile
Enrolled students
339
Military
48%
Average age
35.6
Women
11%
Men
89%
Students of color
35%
A Student-Centered Adult Learning Model
To serve the unique student body, Excelsior College applies a learning model that is specifically designed for adult learners, who want to improve career prospects and expand individual horizons but need a flexible learning format that enables them to study at an individual pace and rate.
This model of education equips successful students to further their careers through enhanced knowledge, understanding, and application of what was learned to their work environments. At the core of this model is advising and student service that enables students to access different forms of learning that meet their individual needs. Another aspect of this system is the use of technology to make distance education not only possible but convenient and accessible, especially for those segments of the population that are otherwise underserved by the mainstream academic institutions (Long et al. 2006).
The differences among the credit sources recognized by Excelsior are self-explanatory as follows:
* Credits from regionally accredited institutions
* Credits for ACE/PONSI approved courses
* Credits through assessed certifications, training, and examinations
* Credits through Extra Institutional Learning processes
* Credits through EC portfolio assessment for prior learning
* Credits from non- regionally accredited institutions
* Credits earned through online courses at Excelsior College and preferred providers
Note that no credit is given for “life experience.”
The ease of integration of credits earned from various sources, absence of residency requirements, along with non-punitive transcripts might be misconstrued to mean that that EC is an easy place to gain a degree. In order to avoid any misconception, Excelsior has developed a system of checks and balances in the form of appropriate and robust policies, procedures, and mechanisms that make the EC an outstanding alternative education provider.
One crucial element in this quality assurance process is the capstone experience/end of program evaluation. This article discusses the instrumental role of the Integrated Technology Assessment, a capstone course offered by the B.S. Information Technology program at Excelsior College and its contributions to the continuous improvement of the program.
2. SPECIALIZED ACCREDITATION
The value of regional and professional accreditation is well established in the educational community and the literature (Anwar et. all, 2005). Accreditation is perhaps even more valuable at a non-traditional institution such as Excelsior. The accreditation process is one very crucial source of input to the institution’s continuous improvement program. It provides a metric against which the institution can assess its performance. It helps assure all stakeholders, students, faculty, and prospective employers, of the value of this form of education. Most importantly, it provides a continuing reminder to the faculty of the professional goals of technology education, and provides a guidepost for the degree of rigor needed in coursework.
The College is accredited by the Commission on Higher Education of the Middle States Association of Colleges and Schools. All its programs are approved by the New York State Education Department and its examinations are recognized by the American Council on Education (ACE). The School of Business and Technology at Excelsior College offers a variety of degree programs, two that are accredited by TAC of ABET (Note: ABET is a non-profit organization that accredits United States postsecondary degree programs in applied science, computing, engineering, and technology.) They are the baccalaureate degree programs in Electronics Engineering Technology and Nuclear Engineering Technology. Adapting the capstone model offered by the two ABET accredited technology programs, the BS Information Technology launched its first capstone course in July 2007. In the following paragraphs, the authors provide an overview of the capstone experience/end of program assessment and how this capstone experience contributes to the continuous improvement of the program.
3. CAPSTONE EXPERIENCE INTEGRATED TECHNOLOGY ASSESSMENT
The capstone experience/end of course assessment serves several critical functions in the process of assessment of the program outcomes. First, it is the assessment tool that directly measures the achievement of student learning outcomes, a vital indicator of a program’s performance. Second, it also serves as a mechanism through which the institution can ensure that all its graduates have passed all the benchmarks for acceptance in their fields. Third, it provides a crucial source of input that drives the continuous improvement of the program (Thinger et. al, 2007). Capstones are cited in the Characteristics of Excellence in Higher Education (2006) as an effective direct measure of the assessment of student learning (p.64).
All BS candidates in the three technology programs, Electronic Engineering, Nuclear Engineering, and Information Technology, must complete an Integrated Technology Assessment (ITA) requirement.
The ITA is a portfolio development process that requires students to reflect on their past academic and professional experiences and use the information gained from this reflective exercise to develop learning statements related to the technology degree outcomes. The learning statements must be supported by documented evidence that demonstrates that the program outcomes have been met.
Students learn how to develop a portfolio during the first few weeks of a 15-week course. They then work under the guidance of a faculty mentor during the remainder of the semester to compose learning statements, compile appropriate evidence, and create the Integrated Technology Assessment portfolio.
The portfolio development process allows the student to take stock of where the student is in his or her career. The faculty mentor guides the student through the ITA process. The process starts with the establishment of an overall plan for the ITA and proceeds through a set of refining stages that lead to a detailed and concrete description of the student’s accomplishments.
Every ABET accredited program offered by the School has 13 earning outcomes. (Note: ABET is a non-profit organization that accredits United States postsecondary degree programs in applied science, computing, engineering, and technology.) Learning outcomes can be divided into two categories as follows:
1. General education outcomes that apply to all programs. Examples include (1) ethical practices, (2) communication skills, (3) working in teams, and (4) commitment to lifelong learning.
2. Discipline specific outcomes that apply to the content of a particular degree program.
The learning outcomes for each program of study have been developed by the School faculty committee that is responsible for the program. Program outcomes are designed to ensure program quality. They also take into account recent developments in the field so that each curriculum remains up to date.
4. IT PROGRAM OUTCOMES
Since the focus of this paper is on IT, we list here the 13 outcomes that are used in the College’s BS/IT program.
1. Apply knowledge of computing and mathematics for problem solving in the field of information technology.
2. Demonstrate the ability to identify and analyze user needs to define and create appropriate computing requirements and solutions.
3. Demonstrate the ability to effectively select, evaluate, and integrate information technologies based solutions in a user environment.
4. Demonstrate the ability to participate effectively in groups or team projects.
5. Demonstrate an ability to understand professional, ethical and social responsibilities, including the impacts of culture, diversity, and interpersonal relations.
6. Demonstrate proficiency in communicating technical information in formal reports, documentation, and presentations to users and informationtechnology professionals.
7. Demonstrate the ability to identify and analyze the impacts of information technologies and computing on public, organizations, and individuals.
8. Demonstrate the ability to identify and apply current and emerging technologies and tools for information technologies solutions.
9. Demonstrate expertise in the core information technologies such as data base management, information management and security, object oriented programming, computer architecture, systems architecture, operating systems, and networking,.
10. Demonstrate the ability to analyze computing and information security requirements and risks, and apply the appropriate tools and techniques to protect organizational data assets in an ethically responsible manner.
11. Demonstrate the ability to apply best practices and standards for information technology applications.
12. Demonstrate the ability to assist in the creation and execution of an effective project plan.
13. Demonstrate a commitment to professional development and to continue to engage in lifelong learning.
5. COURSE STRUCTURE AND ADMINISTRATION
This 15 week course proceeds through a set of activities that starts with the preparation of the student’s professional resume and continues with the development of his or her overall ITA plan. As the course continues, the student develops and refines his or her plan under the guidance of the faculty mentor. Once the final plan is approved by the faculty mentor, each student develops learning statements that pertain to each program outcome and begins gathering supporting evidence for each learning statement.
Everything that the student submits is initially in draft form. The faculty mentor reviews each submission and provides feedback. Once the student has assembled his or her final professional resume and all learning statements and supporting evidence, the student will be in a position to assemble the completed ITA portfolio.
Learning Statements
The real heart of the ITA process is the forming of the learning statements and compiling the supporting evidence. However, first the student needs to understand what constitutes a suitable learning statement.
Definition: A learning statement addresses learning outcomes by including an explanation and example of how the learning outcome has been met. It should include how the student has applied that knowledge to understand the principles in the outcome, how it has helped him or her to solve a problem, and then include evidence to support this. The learning statement should show the relationship of the outcome to the evidence. Learning statements are commonly used across all ABET accredited programs. A good learning statement should:
* Be written clearly and concisely in Standard Written English
* Be written in the form of a narrative
* Reference and provide supporting evidence
* Describe how the evidence supports the learning statement
* Show how the learning statement relates directly to the outcome
* Address each of the characteristics of the outcome in full
Acceptable Supporting Evidence
Now that we have reviewed the definition of a learning statement, we need to know what constitutes acceptable evidence in support of a learning statement and its associated program outcome. For purposes of the ITA, all evidence in support of a learning statement must be in the form of a document. Let us review some typical examples. Examples include a report that the student wrote that demonstrates his or her knowledge or competence; a circuit diagram, systems flow chart, entity-relation diagram or computer source code that the student developed; professional certificates or licenses that the student has earned; completed course assignments, lab reports, a term paper or an exam the student took; a letter of praise from the students manager or professor attesting to the students knowledge, competence or character. It should be emphasized that a college transcript that lists the courses and grades the student has taken is NOT evidence in itself. Let us look at some examples of learning statements from the IT program.
Example 1:
Below is shown an example of a learning statement for program outcome 2, which is:
“Demonstrate the ability to identify and analyze user needs to define and create appropriate computing requirements and solutions.”
Let’s take a look at a learning statement for this outcome used by a past student:
I have demonstrated the ability to identify and analyze, by evaluation, user needs of IT systems. As a Journeyman penetration operator for the NSA Red Team, I created data analysis contributions for major Combatant Commands and Department of Defense (DoD) entities. This analysis provided valuable, technical insight into network vulnerabilities and provided mitigation recommendations to increase security.
To conduct these operations, I used multiple commercially available and NSA proprietary tools and resources that addressed a wide range of IT solutions. I conducted open source research, exploitation, escalation, and on-the-fly and long term analysis of information gathered. From this experience I researched and executed web browsing techniques to disclose unauthorized access, evaluation of commercial and private tools, developed a process to highlight weak operating procedures, and developed technical reports that were delivered to each customer specifically detailing the type of vulnerabilities and counter measures that could be applied.
These operations directly reflect the needs of a government model to have a secure infrastructure and point of presence.
This is a particularly suitable learning statement, since it describes how the student has applied knowledge of the sciences to a real world problem.
The student would then provide documentation of some type that attests to the student’s knowledge and skills in the area referred to in the learning statement. As mentioned before, examples might be a letter from a supervisor, work log describing the tasks performed, or perhaps a report written by the student describing his or her duties.
In the above case, the student submitted a Joint Service Commendation Medal write-up and certificate, and Navy evaluations of this work.
Example 2:
Now let’s look at another learning statement for the same program outcome:
I completed the CompTIA A+ Essentials that measures necessary competencies of an entry-level IT professional with at least 500 hours of hands-on experience in the lab or field. It tests for the fundamentals of computer technology, networking and security, as well as the communication skills and professionalism now required of all entry-level IT professionals. I also completed Systems Analysis I and II, and Database Management Systems at Sinclair Community College. My attached Excelsior College Status Report documents this formal coursework.
Although this looks pretty good at first glance, it is actually a lot weaker than the previous learning statement, since it doesn’t demonstrate what the student knows and is able to do. In fact, by itself it is unacceptable. If, however, the student submits as evidence completed course homework assignments, lab reports, a term paper, a graded exam taken by the student, or similar documentation showing what the student can do, then the learning statement can be used.
So, the key lesson here is for the student to retain all past course homework, term papers, lab reports, and project reports.
Example 3:
Below is a program outcome for the BS in Information Technology:
“Demonstrate the ability to analyze end user data requirements, develop a good logical data design, and implement a working database application that satisfies end user requirements.“
Here’s a learning statements for this outcome used by a past student:
As a systems engineer at IBC Bank, I work with the team of database administrators on a daily basis. All of the systems I support use a backend database. Some of the daily tasks which I work on include daily maintenance items such as backups, reports, and general systems management. Also, many of the systems I support such as the Internet Banking system and the Consumer Lending system require frequent updates to the software. Before I apply any software updates, I perform both system and database backups to facilitate quicker recovery if the update fails for any reason. I also occasionally write queries to extract data from the system to assist business owners in troubleshooting problems or to simply extract data from the logging databases.
This learning statement has some desirable features, since it demonstrates the “ability to analyze end user data requirements,” but is not completely satisfactory. Note that there is no mention of “good logical data design” or implementing “a working database application.” It is certainly usable, but other learning statements will be needed to fully meet the requirements of the program outcome.
Example 4:
Now let’s look at a program outcome that applies to all programs:
“Demonstrate an ability to understand professional, ethical and social responsibilities, including the impacts of culture, diversity, and interpersonal relations.”
Here’s what one student submitted:
I have recently completed a course in business ethics which brought to light the importance of social responsibility, diversity, and ethical business practices. I have used the principles learned in ethics to realize every relationship you have with seniors and subordinates is unique. Sometimes issues aren’t always as simple as they might seem. I have developed myself to view the issues from the other person’s angle; this has developed a strong cohesion among the sailors in my crew.
This is a fine example, since it combines an educational activity and shows how the student has used the knowledge gained in the course to improve the student’s work environment.
Special Circumstances
As we mentioned before, not everyone has the same educational and career background. Let’s look at a couple of special circumstances. Suppose for example that a student has completed his or her course of study, but has no work experience in the field. How does the student go about forming learning statements and compiling supporting evidence?
In this case, the student may describe the course material covered that pertains to a given program outcome and then submit evidence of his or her knowledge and skills in the form of class assignments, exams, projects or perhaps a term paper. As mentioned earlier, letters from the student’s instructors, if they can be obtained, are acceptable items of evidence as well.
Now let’s suppose that the student has received credit by examination, but has not taken a course, and so has no course materials to submit as supporting evidence; and, the student also has no work experience. What do they do now?
This is certainly a more challenging situation, but with effort it can be overcome. Basically, to be able to submit evidence in document form, the student will need to undertake individual work, such as writing a research paper, performing a lab experiment and writing a lab report, building a model or simulation, writing a computer program or other activities that are related to the program outcome that demonstrate his or her knowledge and ability to apply that knowledge. It is not enough for the student to simply send a copy of his or her exam score.
Finally, we require that to receive full credit for demonstrating competence on a program outcome, the student should submit at least three learning statements and associated supporting evidence. This is an example of “multiple measures”, which makes for a more convincing argument that the student has indeed met the requirements of the program outcome. Original documents that the student wishes to keep, such as a letter from a manager or a professional certificate can be scanned and converted to PDF format. A copy of the scanned document is acceptable evidence.
Also, a document can be used in support of more than one program outcome. For example, a project report written by the student could support outcomes related to (1) written communication skills, (2) ability to work well in a team setting, and (3) project management skills.
Course Grading Rubric
The student must satisfactorily address each program outcome by developing appropriate learning statements and providing supporting evidence. The learning statements for each program outcome are graded on the following scale:
* 0 – Not Responsive to Outcome (Evidence not provided for relevant courses or experiences. Coursework and other examples not demonstrative of required knowledge. The student will need to improve the learning statement and/or supporting evidence under the guidance of the instructor.)
* 1 – Minimally Responsive to Outcome (Presents appropriate course evidence with a few examples from coursework and a few connections between coursework and applications)
* 2 – Responsive to Outcome (Presents multiple examples of applications of advanced coursework, on the job, or in other life experiences)
* 3 – Highly Responsive to Outcome (Presents many detailed examples of applications to coursework, job, and other life experiences)
The End Product
Students are given clear instructions regarding the format to be following in the preparation of their ITA portfolio report. The final draft in electronic form is reviewed by the faculty mentor and any remaining corrections are returned to the student for revision.
Once the faculty mentor has given final approval to the ITA portfolio report, each student is responsible for compiling his or her set of learning statements, all supporting documents, a professional resume, table of contents, and cover sheet. The student then places these materials in a three-ring binder, and mails the binder to the School. Students are advised to make a copy for themselves, since the one they send to the School will not be returned.
6. HOW THE ITA HAS WORKED SO FAR
When ITA was first offered in 1997, it was originally offered to the graduate candidates of the baccalaureate degree programs in Electronics Engineering Technology and Nuclear Engineering Technology. Following the successful footsteps of the two ABET accredited programs, the BS in Information Technology program first launched the ITA in July 2007.
Since July 2007, the BS in Information Technology program offered a total of 10 sections of the ITA with 84 students having taken the course with an average passing rate of 90%. The students who have not successfully completed the ITA for the first time are provided with another chance to complete the ITA course. The students must successfully complete the ITA to graduate from the program.
Evolution of ITA
The delivery and design of the ITA has gone through various phases of evolution. The ITA was first implemented as a “do it yourself scrapbook” of the student’s achievements. Based on valuable feedback from students, faculty, industry advisors and consultants, it grew into a more formal experience, with interactive faculty mentoring and a standard grading rubric.
In its present form, the ITA is an online course that not only supports enhanced interactivities between learners and their mentor but also brings a sense of formality in this portfolio development process. Overall, it gives students a sense of greater achievement upon the completion of the course.
Challenges and Strategies
A primary challenge of ITA has been to convince students that ITA is useful and valuable experience for the students. Many students do not find it meaningful to take (and pay for) a course that has no textbook or body of materials to master. Students’ initial reaction to taking this course is simply to fulfill the requirement for graduation.
To improve students’ motivation, a personalized narrated PowerPoint presentation has been created not only to add a personal touch to the course but also to communicate with students about the purpose of the ITA and how the ITA can be used in various career-related situations.
Implementation of the ITA has been one of the most rewarding developments in improving the Excelsior learning model. The ITA has been used as the tool to validate that all the graduates in the technology programs offered by the School of Business and Technology have passed all the benchmarks for acceptance in their fields. It not only gives the student a sense of great achievement, it puts Excelsior’s distinctive stamp on the student’s education, no matter how few or many courses he or she has taken at Excelsior.
7. CONCLUSIONS
The demographics of the students in higher education have been shifting to a more diverse population. The change in the student population calls for more flexible academic programs that can fit various needs of such a diverse population. Excelsior College provides a model of development, implementation, and evaluation of academic programs that is convenient and accessible, especially for those segments of the population that are usually underserved by mainstream academic institutions, and so might never have the opportunity for achievement and recognition under previous college education paradigms.
The ITA plays a critical role in balancing the flexibility and accessibility inherent in this innovative model with rigor and accountability. ITA not only serves as a direct measure for outcome-base assessment, a standard criterion that is required by most of the professional accreditation agencies, but also helps inform and improve the concepts to the continuing benefits of the program and individual students.
8. REFERENCES
ABET (2007) Criteria for Accrediting Engineering Technology Programs. ABET
Acharya, C. (2003) “Outcome-based Education (OBE): A New Paradigm for Learning.” CDTLink, November. Retrieved July 30, 2009 from http://www.cdtl.nus.edu.sg/link/nov2003/obe.htm
Characteristics of Excellence in Higher Education (2006), Middle States Commission on Higher Education, PA.
Evaluation.” Nursing Education Perspectives, Vo. 28 Issue 3, p130-135.
Flood, L. (2007) “Using a Capstone Cultural Diversity Paper for Program Outcome.
Gananapragasam, N. (2008) “Industrially Sponsored Senior Capstone Experience: Program Implementation and Assessment.” Journal of Professional Issues in Engineering Education & Practice, Vo. 134 Issue3, p 257-263.
Long, L., Memon, A., Shih, L., and Tinger, D. (2006) “Nuclear Engineering Technology Students Capstone Experience to Assess the Technical Competencies: A Case Study.” Proceedings of ASEE Annual Meeting, Chicago, IL. . Retrieved September 21, 2009 from
http://soa.asee.org/paper/conference/paper-view.cfm?id=2848
Morris, L. (2005) “Challenges of Access, Affordability, and Persistence.” Innovative Higher Education, Vo. 30 No. 3, p147-148.
The American Association for Higher Education (2005) “Nine Principles of Good Practice for Assessing Student Learning.” Stylus Publishing, LLC.
Thinger, B., Memon, A., and Shih, L. (2006) “Non-Traditional Learning and Assessment Approach to Nuclear Engineering Technology Education.” Proceedings of ASEE Annual Meeting, Chicago, IL. . Retrieved September 21, 2009 from
http://soa.asee.org/paper/conference/paper-view.cfm?id=2167
